artem kirsanov

Refactor these into their own notes…

video notes

• Brain Criticality - Optimizing Neural Computations - Artem Kirsanov
• How Your Brain Organizes Information - Can We Build an Artificial Hippocampus - Artem Kirsanov

Building Blocks of Memory in the Brain

key takeaways:

• a specific set of neurons firing together, is a way how memories get stored in a “memory trace”
• engrams are sparse and non-overlapping between different memories. sparsity differs across brain regions. sparcity rate is unaffected by intensity of stimulus. brain self-regulates sparcity:
• neurons have different excitability levels. (excitability == do they fire easily, requiring little stimulus)
• more excitable neurons are more likely to get recruitet into a memory trace “engram”
• whoa this one is cool: excitatory neurons excite inhibitory neurons, which in turn inhibit other excitatory neurons → they suppress other excitatory neurons in a competiton (e.g. to get selected for the engram)
• blocking inhibitory inter-neurons results in increased engram sizes
• excitability of neurons varies!! else, it would always be the same neurons participating in engrams (not efficient). the excitability sort of oscilates somehow
• memories are not stored in a single region. they are scattered in sparse ensembles of neurons throughout the brain
  • this - yet again - supports total soup theory: there are no special memory-blocks, etc. pp. of course there are specializations, but we can’t hand-craft those into the architecture, hence soup. The dynamics of the system - if set up right - will bring order into chaos
  • this of course does not exclude specializations / enhancements possible with artificial brains: we can hook up a llm or some internet-scale embedding knowledge embedding db to this - and it will learn to work with it, just like any other input stimuli, but way higher bandwidth than e.g. visual or auditory information processing
• neuron-overvlap of two engrams corresponds to memory connectednes / similarity
  • these connected memories can tend to be forgotten / remembered together
• if two memories are related, but this is only discovered later (or e.g. a related memory is made at a potentially much later point int time), meaning they get activated together a lot → strengthening the connections / overlap between them → reorganization, more shared or new neurons, which hold information over the link between the two memories, not the content.

Theta rhythm: A Memory Clock

summary:

• but the medial septum acts as a conductor (“Dirigent”), an upstream oscilator, which provides a rythm “theta rythm” (there are also rythms at other frequencies)
  • it’s active during locomotion, exploratory sniffing, other environment related behaviour (running around, …)
  • this oscilating wave-pattern of voltage arises due to many neurons firing at the same time in rythm
  • this rythmic activity is induced via pacemaker neurons from the medial septum, which sends Hyperprolarization Activated Channel (HCN) proteins to neurons, which allow ions to flow into neurons at regular intervals.
• the hippocampus also has the capability to intrinsic oscilation due to the interplay between inhibitory and excitatory neurons, but it is more limited
• the theta rythm is like an internal memory clock, to which neurons can schedule their firing to an “external” reference signal
• but it also has a second role: the sequential recall of memories
• neurons have a so called “phase field”: a field of input over which they spike the strongest (e.g. certain colors, places, …)

Here you can see the activation of a single place cell, spiking earlier and earlier in the phase as its location is traversed:

A spike at 180° of the wave means that we are at this moment at the exact location of the place cell. This way, information about sequential order of memory patterns (here different space-cells in the diffrerent colors) is encoded over space / time:

For soup, ig we would have different phase cycles as inputs (just like the reward center).
Let a certain fraction of neurons connect to it (or yk not a hand-picked fraction, but the # connections is limited by constraints).
Idk if we would distribute input nodes uniformly accross the soup? Most likely not, as, as far as i understood, at least with theta waves, they are mostsly used for hippocampal place cells. So it miiight make sense to dig deeper ¹ into details like which exact roles theta rythm plays where (and accordingly place other signal sources in the soup), esp. for early prototypes where we don’t have $\infty$ compute.

Logarithmic nature of the brain

lognormal distribution

A lognormal distribution is a statistical distribution characterised by a skewed bell-shaped curve. It arises, for instance, when taking the exponential of a normally distributed variable. It differs from a normal distribution in several ways. Most importantly, the curve of a normal distribution is symmetric, while the lognormal one is asymmetric with a heavy tail.

 The lognormal distribution arises naturally as a result of multiplicative processes, similarly to how the normal distribution emerges when many independent variables are summed.

$$\log (a\cdot b)=\log (a)+\log (b)$$

See also Logarithm turns multiplication into addition & Why we want additive instead of multiplicative

Examples of lognormal distributions

• synaptic activity
• synaptic weights
• the number of connections a neuron has
• neuron firing rates
• the size of files on a computer
• the length of words in a language
• the size of living organisms
• the distribution of income
• the size of cities
• the size of companies

Link to original

Firing rates of neurons are distributed on a log-normal scale:

• The majority of neurons is slow-spiking (at ~1Hz)
• About 10% of neurons is fast-spiking (at ~10Hz)
  Same with synapses:
• Most synapses are quite weak, but
• there are a couple of very strong synapses.
  Same with spatial distribution of neurons in different regions of the cortex.

Strong synapses == more impact on receiving neuron, more reliabe communication highways.
Neurons with a high firing rate also tend to make stronger connections to eachother.
These neurons also tend to grow thicker axons == faster signal transmission.
These 10% of strong, fast firing neurons account for ~50% of information sent in the brain.
The FF minority has access to lots of information, with which they can draw more general conclusions by treating different inputs simmilarily (e.g. “we are in a room now”), wheras the slower specialists will determine exact features.
These properties are smoothly distributed on the log-normal spectrum (again, a property that has to emerge).

Change in spine size is proportional it its size

$$\frac{dN}{dt}\propto N$$

This explains the multiplicative dynamics / log-normal distribution of synaptic connection strength.
However, we don’t know yet how these dynamics arise for the other mentioned properties.

Some commenter:

Log-normal distributions are closely related to pink noise (power is 1/freq), since d(log) = 1/x. This is said to be the hallmark of self-organization. It shows up everywhere you have fractal symmetry: brains, turbulence, finance, weather, even migration patterns

Memory Consolidation: Time Machine of the Brain

The cortex is the executive center - the executive center of the brain - where thoughts, memories and language reside.
The hippocampus is a loopy structure around the cortex with mainy interconnections to it. It Sort of manages the information in the cortex, a bit like a librarian.

During wake times, mostly the cortex is firing, but during sleep this is reversed.
But the firing happens in “sharp wave ripples”. Sudden spikes, oscilation at the peak and then a quick decay.
These firings have been shown to consist of place cells that fired during the day firing in reverse order and on a smaller time-scale “hippocampal fast reverse replay”.
This is not just exclusive to sleep, but also active when a mouse is resting for example.
It is also used not just for recapping, but also for planning through future actions, as those waves also occur - in the correct order! - when a mouse decides which path to take for example.
So pretty much just world-model vibes.

The Physics Of Associative Memory

See Hopfield Network

References

fractal
neuroscience
soup
People

Footnotes

1. However, similar phenomena have been observed in other types of neurons as well. For instance, grid cells in the medial entorhinal cortex, which fire in multiple locations that form a grid-like pattern in space, also show phase precession. Furthermore, there is evidence for phase precession in other brain areas and for other kinds of coding, such as in the prefrontal cortex where neurons may show phase-locked firing to cognitive tasks that are not necessarily spatial. - chatty

   ↩