options framework

In standard RL, actions are single-step: observe, act, observe again. Options extend this to temporally extended actions. An option like “navigate to the hallway” runs its own internal policy for many steps before terminating and handing control back.

Option (Sutton, Precup & Singh, 1999)

A 3-tuple $(\mathcal{I},\pi ,\beta )$:
$\mathcal{I}\subseteq \mathcal{S}$ … initiation set, states where the option can start
$\pi :\mathcal{S}\times \mathcal{A}\to [0,1]$ … intra-option policy, how to act while the option runs
$\beta :\mathcal{S}\to [0,1]$ … termination condition, probability of stopping in each state

Available in state $s$ iff $s\in \mathcal{I}$. Once initiated, actions follow $\pi$ until the option terminates stochastically according to $\beta$.

A primitive action is a degenerate option: $\beta (s)=1$ everywhere, terminates after one step. An agent with options learns a policy $\mu :\mathcal{S}\to \mathcal{O}$ over the set of options $\mathcal{O}$ rather than over primitive actions directly.

Because options have variable duration, the resulting decision process is a semi-MDP: transitions take variable amounts of time. The Bellman update generalizes to a multi-step TD target. For an option lasting $\tau$ steps:

$$r_{t+1}+\gamma r_{t+2}+\dots +{\gamma }^{\tau -1}{r}_{t+\tau }+{\gamma }^{\tau }\underset{o}{\max }Q(s_{t+\tau },o)$$

The discounted rewards collected while the option ran, plus the discounted value of the state you land in. Same structure as a regular TD update, just spanning $\tau$ steps instead of one.

In practice, options are often implemented by expanding the action space. If the original game has $|\mathcal{A}|$ actions and you offer $|\mathcal{T}|$ possible durations, the agent picks from $|\mathcal{A}|\times |\mathcal{T}|$ options (e.g. “move left for 4 steps”, “fire for 1 step”). The intra-option policy is trivial here (repeat the chosen action), but the framework is general: $\pi$ can be an arbitrary policy, like a learned sub-network that handles a whole subtask.

Automatically discovering good options (reusable sub-behaviors that transfer across tasks) is a core problem in hierarchical RL.