information theory

Motivation

Approaches to understand digital brains:

• observe external behavior
• analyze structure
• knockout experiments (structure + behavior)
• observe internal behavior (state-to-state)
• use information theory (quantitative analysis of behavior)

Introduction to Shannon Information Theory

Claude Shannon was interested in how to efficiently and accurately transmit information by mathematically modelling communication, focusing on both compression (efficiency) and error correction (accuracy).
We can leverage Shannon’s work on compression as a tool to identify and quantify informatino in complex systems, such as ALIFE.

We use the unit of bits in SIT.
1 bit is 1 yes/no question, which is a very intuitive measure.
You could use base 10, or e or anything else, but why would you.

Think about a random process that generates ordered sets of data.
This can be related to any indexable property (time, position, altitude, temperature, …)
This random source of events draws from an alphabet of symbols. Each symbol is called a token.

Surprise is a measure of how unexpected an event is, quantified in bits.
It tells us how much we learned from an event.
If an event A has a surprise of 2 bits, you learned 2 yes/no questions.

Surprise relates to symbols of the process, while entropy relates to the process itself.
Entropy is just the expected value of surprise.

$$H(X)=\sum _{x\in X}P(x)I(x)=-\sum _{x\in X}P(x){\log }_{2}P(x)$$

where $P(x)$ is the probability of event $x$, its “relevance”, and $I(x)$ is the surprise of event $x$.
If a process has 1 bit of entropy, then on average, you need just 1 question to know the behavior.

A fair coin flip has exactly 1 bit of entropy.
The AND function has 0.8 bits of entropy, since we expect false to be more likely for random inputs.

Todo