year: 2021/07
paper: https://direct.mit.edu/isal/proceedings/isal2021/33/116/102923
website:
code: https://colab.research.google.com/github/google-research/self-organising-systems/blob/master/notebooks/async.ipynb
connections: NCA, asynchronicity, Eyvind Niklasson, Alexander Mordvintsev, Ettore Randazzo
Re-read and take notes after optimization bootcamp, & deeper laplacian, PDE, ODE solver understanding.
TLDR
Neural Cellular Automata (NCA) have been applied to a diverse set of tasks, and have shown a propensity to learn complex behaviour despite small model sizes. In a biological setting, similar tasks must be performed in the presence of noise, both in the environment and internal to the execution of the cell’s functions, and without a global clock. Thus, one would expect a biological model to be robust, self-correcting and impervious to order of execution among cells.
The training regime for NCA is straightforward and the underlying models are often quite small by the standards of deep learning today (on the order of 1e3 parameters). The authors’ use of asynchronicity in the update step is not introduced as an attempt to improve convergence, but rather to ensure the underlying model operates in a completely local fashion.
Synchronous updates in a system generally require a shared global clock. In a biological system or artificial distributed system, this is not always feasible or desired. Synchronicity can be achieved through clock synchronization algorithms in-silico, or signals globally diffusing over cells clusters biologically, but a preferable solution is for cells to themselves learn an optimal version of such behaviour or to operate in a fashion that is largely invariant to global time.
Contribution
Mordvintsev et al. (2020) note that NCA learn a dynamical system, and Niklasson et al. (2021) make this link more clear by framing the training of NCA as both learning and numerically solving a partial differential equation. We expand on the analogy and discuss some of the subtleties as-sociated with it. We demonstrate that asynchronicity during training not only satisfies an ideological requirement of the NCA to operate in a truly local fashion, but also makes the learned PDE more well behaved, allowing the model to be robust to altern
