policy

Types of policies

Deterministic policy: $a_t=\mu (s_t)$

Stochastic policy: $a_t\sim \pi (\cdot \mid s_t)$

Two major types are categorical policies for discrete action spaces and diagonal gaussian policies for continuous action spaces.

Categorical Policy

A categorical policy is just a classifier over discrete actions.
So like usual you have feature extraction → logits → softmax → sample an action.
Get the log-likelihood for the action $a$ at state $s$ by indexing the log of the output vector: $\log {\pi }_{\theta }(a|s)$.

Diagonal Gaussian Policy

Diagonal gaussian policies map from observations to mean actions ${\mu }_{\theta }(s)$.
There’s two common ways to represent represent the covariance matrix:

A parameter vector of log standard deviations $\log {\sigma }_{\theta }$, which is not a function of state.

Network layers mapping from states to log standard deviations $\log {\sigma }_{\theta }(s)$, may share params with ${\mu }_{\theta }$.

We can then just sample from the distribution to get an action.

Log standard deviations are used so we don’t have to constrain the ANN output to be nonnegative, and can simply exponentiate the log outputs to obtain $\sigma$, without loosing anything.