Actor critic methods are reinforcement learning algos with two learned parts:
- actor: the policy ; it acts (picks actions).
- critic: a learned value function (a separate network or a head on the actor); it critiques those actions by estimating expected return. Usually the critic is the state-value (as in A2C/A3C, PPO); in the off-policy continuous-control family (DDPG/SAC/TD3) it’s the action-value .
Actor critic addresses the bias-variance tradeoff: Instead of weighting the policy gradient by the raw, high-variance sampled return (as in REINFORCE), we weight it by an advantage built from the critic, far lower variance, at the cost of bias from the critic’s approximation error:
Link to originalMotivation
In pure vanilla policy gradient, we use the actual (empirical) returns , which is unbiased, but high variance: empirical returns from the noisy environment and stochastic policy, the variance of the gradient estimator scales unfavorably with the time horizon, since the effect of an action is confounded with the effects of past and future actions.
actor critic methods alleviate that by estimating a value function instead, at the cost of introducing bias (see bias-variance tradeoff).
→ While variance is annoying as it makes your training potentially unstable and requires more examples to train on, bias can be a bigger problem, as you model might learn the wrong thing, and even with infinite samples never find the optimal policy (fail to converge, or converge to a suboptimal solution).
The actor follows the policy gradient, weighted by the critic’s advantage estimate.
The critic is regressed toward its own value targets; either TD targets (bootstrapping) or monte carlo returns. GAE balances the two.
Because each learner chases a target the other keeps moving, actor-critic is prone to instability, hence tricks like target networks, trust regions (TRPO/PPO), and careful learning-rate balancing.