year: 2020
paper: https://arxiv.org/abs/2007.00970
website:
code:
connections: meta learning, message passing, self-organization, MAML, NCA, Ettore Randazzo, Eyvind Niklasson, Alexander Mordvintsev
For the most part, learning algorithms have been hand-crafted.
The deep learning community has largely converged to use almost exclusively gradient-based approaches for learning a model’s parameters. Such gradient-based approaches generally impose limitations in terms of the loss landscape, choice of network architecture and training dynamics. A non-exhaustive list of examples is: their inherent tendency to overfit to training sets, their catastrophic forgetting behaviour, their requirement of a smooth loss landscape, and experiencing vanishing or exploding in recurrent or large architectures. Moreover, while the mechanics of artificial neurons are inspired by their biological counterparts, they have been greatly simplified to be scaleable and differentiable, rendering them far less powerful than their biological counterpart. Perhaps the simplicity of its building blocks is the reason of why most of deep learning research occurs on layered deep networks that require increasing amounts of computation and memory, limiting the explorations on fundamentally different architectures. Nevertheless, backpropagation is still the best tool in our toolkit for optimising models with an extensive set of parameters.
Todo