factorized representations enable compositional generalization

factorized representations enable compositional generalization
good representations enable simple methods

It is enough to know the two factors seperately to predict the joint distribution!

→ Allows to predict values for combinations which you have not encountered before
→ Requires much less storage: $O(n)$ instead of $O(2^n)$.

Factorization in the brain

The brain applies factorization in many areas, one example is hippocampal remapping.
Factorizing environments (or all kinds of concepts - in the neocortex) into building blocks allows to easily build new models on the fly.

Engineering example: GLU

---

How Your Brain Organizes Information - Can We Build an Artificial Hippocampus - Artem Kirsanov