Todo
https://www.semanticscholar.org/paper/A-possibility-for-implementing-curiosity-and-in-Schmidhuber/2980dfe5c99658dc3e508d9d6e1d7f26e6fc8934#citing-papers read this paper + go through the citations
Link to originalWhy curiosity?
→ Curiosity is a primary drive in most, if not all, animal behavior.
→ Curiosity is as strong, if not sometimes stronger, than the drive for reward.
→ Curiosity as an algorithm is highly effective at solving difficult optimization problems.The benefits are an optimal value solution to the exploration versus exploitation trade-off. A solution which seems especially robust to model-enviroment mismatch. At the same time curiosity-as-exploration can build a model of the environment useful for later planning, creativity, imagination, while also building diverse action strategies.
Interest / curiosity is an aid to continually improve the compression capability of the brain. It is the extent to which data will change the number of bits needed to encode data (i.e. its beauty) for a subjective observer :
The curiosity reward is then simply: , where is beauty.
In other words: Seeking new, “relevant” information, that leads to (new) higher level concepts, connections etc. → exploring parts of the space where you know that your world model (compressor) is inaccurate.
Self-introspection
In order for an agent to improve its knowleddge about the world - e.g. via delayed reward - the agent needs to model its own ignorance, i.e. a rudimentary form of self-introspective behaviour.
In order to know what you don’t know, you have to get to know yourself.If the learning algorithm depends on the model network, the model network has to make a prediction about its own current prediction capabilities. The activations of the model network are (partly) interpreted as a statement about the current weights of the model network.
White noise and constant input:
This definition also explains why a completely black room is boring: Every next frame is identical to the previous, it is unsurprising. There is no compression progress.
Similarily for white noise: This has maximum shannon information (surprise), again, not compressible, boring.
Notice also the similarities to open-ended.
Transclude of A-possibility-for-implementing-curiosity-and-boredom-in-model-building-neural-controllers#^7328c9
Transclude of A-possibility-for-implementing-curiosity-and-boredom-in-model-building-neural-controllers#^efbff7
Beauty and interestingness are closely related to open-ended.
Papers
The gratification of curiosity rather frees us from uneasiness than confers pleasure; we are more pained by ignorance than delighted by instruction. (Johnson, 1751)
References
Large-Scale Study of Curiosity-Driven Learning
human intelligence