novelty

Novelty

Novelty is transition data that looks different from what an agent has previously experienced, which may be computed on an episodic or lifelong basis.

Measures for novelty

A common measure of novelty for state $s$ is based on methods for counting state visitations, whereby less visited states are consider more novel.

In practice, there are many proxy measures of novelty, like epistemic uncertainty, which may be estimated by the variance of a prediction network ¹.

Similarly, prediction error or regret can also be used as measures of novelty, assuming the agent fares worse on environments that present novel challenges.

Importantly, when applied to the mean performance over a batch of trajectories, these latter approaches implicitly subtract away sources of inherent, irreducible uncertainty that can act as stochastic traps for a novelty-seeking agent.

Alternatively, such irreducible uncertainty can be removed from the calculation by explicitly modelling it. ²

Footnotes

1. Self-Supervised Exploration via Disagreement | Planning to Explore via Self-Supervised World Models | ^77bbaa ↩

2. Source for the quoted note: General intelligence requires rethinking exploration ↩