hebbian learning

When an axon of cell A is near enough to excite cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased. - Donald Hebb

Neurons that fire together wire together.

This statement from Donald Hebb has been interpreted as meaning strengthening of synapses, but it can also mean creation of new synapses!

Hebbian weight update rule

$$\Delta w_{ij}=\eta x_i{x}_j$$

$\Delta w_{ij}\dots$ change in the synaptic weight between neuron $i$ and $j$.
$\eta \dots$ learning rate (small positive constant).
$x_i,x_j\dots$ activations of the pre- and post-synaptic neurons.

It is used for example in Hopfield Network and other associative memory models.

Oja's Rule

A normalized version of Hebbian learning, which addresses the issue of unbounded growth of synaptic weights:

$$\Delta w_{ij}=\underset{\text{Hebbian growth}}{\underbrace{\eta x_i{x}_j}}-\underset{\text{self-limiting shrinkage}}{\underbrace{\eta x_i^2{w}_{ij}}}=\eta x_i(x_j-w_{ij}{x}_i)$$

With standardized inputs, Oja puts weights into the correlation range $[-1,1]$.

Averaging over data:

$$\mathbb{E}[\Delta w_{ij}]=\eta (\underset{C_{ij}}{\underbrace{\mathbb{E}[x_i{x}_j]}}-\underset{v_i}{\underbrace{\mathbb{E}[x_i^2]}}{w}_{ij}).$$

For small $\eta$ (stochastic approximation), the weights follow the ODE

$${\dot{w}}_{ij}=\eta (C_{ij}-v_i{w}_{ij}).$$

A fixed point of this dynamics satisfies ${\dot{w}}_{ij}=0$, i.e. the mean change is zero:

$$0=C_{ij}-v_i{w}_{ij}\Rightarrow w_{ij}^{\star }=\frac{C_{ij}}{v_i}.$$

Interpretation: each synapse converges to “correlation divided by presynaptic variance” — i.e., the regression coefficient of $x_j$ on $x_i$. The decay prevents blow-up and scales the weight to a finite, data-determined value.

Another version is spike-timing dependent plasticity.

Transclude of principal-component-analysis#^852d71