TTS

Growing Artificial Neural Networks for Control: the Role of Neuronal Diversity

Eleni Nisioti${}^{1,†}$, Erwan Plantec${}^{1,†}$, Milton Montero, Joachim Winther Pedersen, Sebastian Risi
{enis, erpl, mile, jwin, sebr}@itu.dk,
IT University Copenhagen
Copenhagen, Denmark

arXiv:2405.08510v1 [cs.NE] 14 May 2024

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
GECCO ‘24, July 14–18, 2024, Melbourne, VIC, Australia
© 2024 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-0494-9/24/07.
https://doi.org/10.1145/3638529.3654113

Figure 1: Neural Developmental Programs with Lateral Inhibition. (A) Growth starts with a minimal number of cells with unique intrinsic hidden states (indicated with numbers) and random extrinsic hidden states (indicated with colors). At each step, each cell independently decides whether to differentiate (update its extrinsic hidden state based on its environment), generate a new cell that inherits its intrinsic state and update a connection. Afterward, the grown network is mapped to a recurrent neural network used to control an agent in an environment. (B) Lateral inhibition: each time a cell differentiates, grows a new cell or updates a connection, it inhibits its neighbors for a fixed number of steps. Inhibition helps maintain diversity in the intrinsic hidden states.

ABSTRACT

In biological evolution complex neural structures grow from a handful of cellular ingredients. As genomes in nature are bounded in size, this complexity is achieved by a growth process where cells communicate locally to decide whether to differentiate, proliferate and connect with other cells. This self-organisation is hypothesized to play an important part in the generalisation, and robustness of biological neural networks. Artificial neural networks (ANNs), on the other hand, are traditionally optimized in the space of weights. Thus, the benefits and challenges of growing artificial neural networks remain understudied. Building on the previously introduced Neural Developmental Programs (NDP), in this work we present an algorithm for growing ANNs that solve reinforcement learning tasks. We identify a key challenge: ensuring phenotypic complexity requires maintaining neuronal diversity, but this diversity comes at the cost of optimization stability. To address this, we introduce two mechanisms: (a) equipping neurons with an intrinsic state inherited upon neurogenesis; (b) lateral inhibition, a mechanism inspired by biological growth, which controlls the pace of growth, helping diversity persist. We show that both mechanisms contribute to neuronal diversity and that, equipped with them, NDPs achieve comparable results to existing direct and developmental encodings in complex locomotion tasks.

KEYWORDS

Evolution, neuroevolution, morphogenesis, development

ACM Reference Format:

Eleni Nisioti${}^1$, Erwan Plantec${}^1$, Milton Montero, Joachim Winther Pedersen, Sebastian Risi. 2024. Growing Artificial Neural Networks for Control: the Role of Neuronal Diversity. In Genetic and Evolutionary Computation Conference (GECCO ‘24), July 14–18, 2024, Melbourne, VIC, Australia. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3638529.3654113

Biological neural networks, a family that exemplifies some of the most complex systems in life, are the products of a growth process that starts with a handful of cells. To support complex behavior, these cells proliferate, differentiate and connect with others, forming networks consisting of thousands of different cell types [15, 17]. They do so with limited information. Genomes are not large enough to directly encode brain states [7, 24] and cells can only perceive information locally. Artificial neural networks (ANNs), on the other hand, follow an engineering rather than a biological paradigm: the architecture and weights are optimized directly and transferred intactly to the next generation [14, 20, 21].

Inspired by the genotype-to-phenotype mapping found in nature, on the other hand, artificial developmental encodings optimize ANNs in a search space of lower dimensionality and define a mapping from this space to the ANN realisation [5, 13, 16, 19]. This process of retrieving the ANN from can be seen as a model of biological development. Some developmental encodings abstract away the temporally-extended nature of development and formulate it as a single-step, non-linear mapping [5, 19], while others explicitly model the self-organised nature of growth using primarily Cellular Automata [12]. Reported benefits of such developmental encodings

are: a) improved generalisation due to the regularising efffect of a genomic bottleneck [7, 19, 24] b) structural patterns such as modularity and regularity that naturally emerge from self-organised processes and can improve, among others, the energy efficiency of phenotypes [19] c) robustness to perturbations that may occur during growth when the latter is self-organised [12].

In this work we study Neural Developmental Programs (NDPs), a previously proposed algorithm for growing ANNs [13] that control the behavior of an agent in RL tasks. NDPs determine both the architecture and weights of the ANN through a temporally-extended, self-organised growth process. Inspired by biological growth, they equip cells with the ability to differentiate, grow new neurons and update their synaptic weights. All cells share a model that determines how these decisions are made based on locally available information. Compared to morphogenetic algorithms that leverage a target at each step of growth [12], the RL setting introduces additional challenges to the stability and success of development: the growth process does not have access to a target control policy but only receives some feedback regarding its performance. Compared to one-step indirect encodings for control [5, 19], NDPs employ a more complex genotype-to-phenotype mapping that increases training difficulty, but introduces the potential of leveraging the benefits of self-organisation for control.

We empirically identify a challenge with training NDPs: if all neurons differentiate into the same type, then growth-related decisions will be identical. ANNs grown by such a naive process have a homogeneous structure and cannot exhibit complex behavior. Based on this observation we propose two modifications. First, employing intrinsic states that are not modified during growth ensures that diversity cannot disappear in the population. As networks are initialized with a small number of cells that each have a unique intrinsic state, some diversity is introduced at the beginning of growth. As the networks grow, these unique states are copied around so that not every cell has a unique intrinsic state. Thus, similarly to biological networks, lineages of cells are formed during growth. As we show empirically, the introduction of intrinsic states can make the difference between complete failure and success in locomotion tasks. Second, we introduce lateral inhibition, a mechanism hypothesized to play an important role in the maintenance of diversity in biological growth [1, 3, 15]. Under this mechanism, when a cell decides to perform an action it inhibits other cells in its neighborhood from performing a similar action for a limited number of steps. Our simulations indicate that inhibition helps maintain neuronal diversity both during growth and during evolution, even in the absence of intrinsic states.

1 METHODS

1.1 The Neural Developmental Program

We model growth as a stochastic graph generation process whose final output is the ANN used to control the actions of an RL agent. At each growth step the state of the agent is represented by a directed graph $\mathcal{G}$G where vertices $\mathcal{V}$V represent cells and edges $\mathcal{E}$E connections between them. Cells and edges are characterized by real-valued vectors, termed their hidden states ($h$h and $e$e for cells and edges respectively). The hidden state of a cell consists of two parts: $h_{\text{ext}}$h extrinsic is the extrinsic hidden state, influenced by its environment, and $h_{\text{int}}$h intrinsic is the intrinsic hidden state that only depends on its lineage (to be defined shortly).

Growth starts with a graph that contains a number of cells equal to the size of the observation plus action space of the environment, as this is the minimum number of neurons that the policy network needs to have. Each cell in this initial graph has a unique intrinsic state. At each step a cell can perform three actions:

• differentiate, i.e., predicts its next hidden state using a model that takes into account the local environment of the cell. We refer to this model as the DiffModel and implement it as a Graph Attention Network (GAT) [22] that takes as input the hidden states of all nodes in the graph and the current connectivity. In general different implementation of the DiffModel are possible, such as graph convolution employed in the original introduction of NDPs [13].
• grow, i.e., generate a new cell. To decide whether to grow a cell employs another model, the GenModel $G(h)\to \{0,1\}$G of h, which maps to zero or one that we implement as a feedforward network shared across neurons. The newly generated cell forms a connection with its parent and inherits its intrinsic state. Thus, lineages of cells are formed during growth.
• update the weight of a connection. The new value of a weight depends on the hidden state of the two cells connected to it and is predicted by a third model, the EdgeModel $E(h_1,h_2)\to \mathbb{R}$E of h one and h two, which maps to a real number, that is also a feedforward network.

Thus, the growth process is orchestrated by the DiffModel, GenModel and EdgeModel, which can be seen as the developmental program that runs in each cell of an organism. At the end of the growth process, which lasts for a fixed number of steps, the ANN has acquired its final connectivity. To derive the behavioral policy we interpret the cells of the final graph as neurons, the weights of the edges as the weights of the ANN, and equip neurons with fixed activation functions to form a recurrent ANN. We present a schematic of our model in Figure 1. To optimize our model for a given task we train the weights of the three models in the NDP using an evolutionary strategy. Thus, the number of optimization parameters does not depend on the size of the control policy.

1.2 Maintaining neuronal diversity

Biological growth is characterized by a diversity paradox: a handful of identical cells grow to become a network consisting of thousands of different neuron types [15]. If neurons change only based on local information and if they all start off with the same environment, then where does this diversity come from? As we show in our experimental analysis, the same question becomes relevant when growing ANNs. Although the DiffModel could theoretically learn to differentiate neurons so that their hidden states remain diverse, this seems hard to achieve in practice. Rather, the hidden states converge to identical values early in the growth process, leading to degenerate solutions where all weights explode to large values or effectively disappear.

To avoid this instability we turn towards a mechanism hypothesized to play an important role in maintaining diversity in biological growth: lateral inhibition [15]. Under this mechanism, a cell that undergoes a certain change, such as differentiation, neurogenesis or synaptogenesis, emits a signal to its neighborhood that prohibits

other cells from undertaking a similar action for a fixed number of steps [1, 3, 15]. This ensures that cells do not make decisions simultaneously. We visualize the concept of of lateral inhibition in Figure 1. As we show in our experiments, both lateral inhibition and the use of intrinsic states contribute to the maintenance of diversity.

2 RESULTS

Experimental setup. We test the ability of NDPs to learn how to solve complex behavioral tasks. In particular, we experiment with the Reacher, Inverted Double Pendulum, Halfcheetah and Ant tasks using the Mujoco Brax library [4]. The three latter tasks evaluate for locomotion while Reacher in addition evaluates the ability to locate different targets. The intrinsic hidden states are one-hot encoded vectors of length equal to the number of initial cells. The extrinsic hidden states are real-valued vectors of length 8. The EdgeModel is a feedforward network with two hidden layers of 16 neurons each with a ReLU activation function for hidden neurons and a linear activation function for output neurons. The GenModel is a feedforward network with one hidden layers of 32 neurons with a ReLU activation function for hidden neurons and a tanh activation function for the output neuron. The target network used to control the policy employs a ReLU activation function for hidden neurons and a linear one for the output layers. Growth lasts for 15 steps and inhibition lasts for 2 steps. All methods are trained with DES [10], which we empirically found to perform better than other evolutionary strategies, using the Evosax library [9]. Results are averaged across 3 training trials and 10 evaluation trials. We provide our code for running end-to-end training on a GPU and visualizations of the learned trajectories in an online repo.

Our main result is that the NDP with intrinsic hidden states discovers optimal policies for all tasks. We compare its performance to two baselines: a) a direct encoding where DES directly optimizes the weights of an RNN with 100 neurons (which corresponds to the maximum size of networks grown by the NDP) b) an indirect encoding that employs an EdgeModel to predict weights based on the intrinsic state of each neuron in a single step. This method can be seen as a Hypernetwork [5] with a pre-determined (rather than learned) encoding of target neurons. Thus, this ablation differs from our NDP in that there are no extrinsic hidden states, the network architecture is fixed and all neurons have a unique intrinsic state. Our objective is to show that the NDP performs comparably to the indirect encoding despite the training difficulties we have discussed. The direct encoding is provided as a baseline that is known to perform well in the tasks we examine and, therefore, serves as an expected upper threshold for fitnesses. As these tasks do not pose any particular need for generalisation or robustness we do not expect to see indirect encodings outcompeting direct ones.

Figure 2 presents the evaluation performance during training, where every 10 generations we perform 2 evaluation trials of the currently best individual. We observe that the NDP performs comparably to the indirect encoding, as ANOVA indicates that differences are statistically insignificant for all tasks. In contrast, evaluating the NDP without intrinsic states led to trivial policies that did not show any improvement in any of the tasks. The direct encoding converges more slowly than the other methods for some tasks (Reacher and Ant), arguably due to the larger size of the search space. In the Ant and Halfcheetah, the direct encoding achieves high speed much quicker than the indirect encoding and NDP, which progress slowly but eventually solve the task (in an additional experiment we saw that the NDP requires about 10000 generations to achieve a fitness of 6000 in the Ant). A possible explanation for this is that Halfcheetah and Ant have a larger action space compared to the other tasks and may, therefore, prove more challenging for indirect encodings or require retuning of the EdgeModel.

In Figure 3 we monitor neuronal diversity both during evolution (where we only look at the final growth step) and during growth (after convergence at the evolutionary scale). Neural diversity is defined as the average distance between a neurons’ hidden states and the ones of its $k=10$k equals ten nearest neighbors. We compare two conditions: NDP with lateral inhibition and NDP without lateral inhibition. In both cases, we remove intrinsic states as they may affect neuronal diversity. We observe that inhibition helps maintain neuronal diversity at both the evolutionary scale and developmental scale while it completely disappears within the first growth steps in the absence of inhibition.

3 DISCUSSION

We studied Neural Developmental Programs, an algorithm inspired by biological growth and models the growth of an ANN as a stochastic graph generation process where decisions are made locally by interacting neurons. We empirically showed that such algorithms are prone to training instabilities, which may explain why they have not been successfully applied to complex RL tasks in the past. We identified neuronal diversity, the presence of neurons with different hidden states during growth, as an important step towards avoiding the degeneration of control policies. To ensure neuronal diversity we introduced two key ingredients: a) neurons start out with unique intrinsic hidden states that are inherited upon neurogenesis b) lateral inhibition is employed when neurons differentiate and generate new neurons. Our additions to the NDP enabled it to go beyond the simple navigation and control tasks investigated in [13] to complex 3D locomotion and manipulation tasks, demonstrating the important role of neural diversity in neural growth.

Our work has been concerned with how to grow ANNs. Another, equally urgent, question is why grow them. Growth can be seen, especially in an engineering-focused deep-learning era, as a limitation rather than a beneficial feature of biological organisms. Yet biologists, complexity- and computer science researchers have voiced a number of hypotheses on how growth can benefit an evolving system [6, 8, 15]. Arguably the benefits of growth become more obvious once one introduces environmental feedback. Then, similarly to developmental algorithms like Hebbian [18] and reinforcement learning, growth can render the organism adaptable to its environment. Could growth be beneficial even in the absence of environmental input? A relevant hypothesis is that evolution faces an upper threshold in the complexity of phenotypes it can create and that growth is a necessary ingredient for evolving systems of certain behavioral complexity [6]. Our current empirical analysis does not show any benefits of temporally-extended growth over the single-shot baseline. We believe that such benefits for complexity and generalisation will appear if one scales up the tasks

Figure 2: Training curves for different control tasks and methods. NDPs perform comparably to the other indirect encoding.

Figure 3: Effect of inhibition on neuronal diversity in the Reacher during evolution (left) and during growth (right).

appropriately. By stabilizing the training of NDPs to work in high-dimensional, continuous action spaces our work lays the ground for such studies. As the design space of NDPs is large, future work can investigate the effect that different implementations of differentiation, neurogenesis and synaptogenesis have on the training stability of NDPs and leverage additional mechanisms and biases from the study of growth in biology and computer science [2, 11, 23].

ACKNOWLEDGMENTS

This project was supported by a European Research Council (ERC) grant (GA no. 101045094, project “GROW-AI”).

REFERENCES

• [1] Jessica L. Ables, Joshua J. Breunig, Amelia J. Eisch, and Pasko Rakic. 2011. Not(ch) just development: Notch signalling in the adult brain. Nature Reviews Neuroscience 12, 5 (May 2011), 269–283. https://doi.org/10.1038/nrn3024
• [2] Sarthak Chandra, Mikail Khona, Talia Konkle, and Ila R. Fiete. 2024. Self-organized emergence of modularity, hierarchy, and mirror reversals from competitive synaptic growth in a developmental model of the visual pathway. https://doi.org/10.1101/2024.01.07.574543 Pages: 2024.01.07.574543 Section: New Results.
• [3] Tarec Fares and Armen Stepanyants. 2009. Cooperative synapse formation in the neocortex. Proceedings of the National Academy of Sciences 106, 38 (Sept. 2009), 16463–16468. https://doi.org/10.1073/pnas.0813265106 Publisher: Proceedings of the National Academy of Sciences.
• [4] C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. 2021. Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation. http://github.com/google/brax
• [5] David Ha, Andrew Dai, and Quoc V. Le. 2016. HyperNetworks. arXiv
• [6] Stuart A. Kauffman. 1993. The Origins of Order: Self Organization and Selection in Evolution. Oxford University Press.
• [7] Alexei Koulakov, Sergey Shuvaev, Divyansha Lachi, and Anthony Zador. 2022. Encoding innate ability through a genomic bottleneck. https://doi.org/10.1101/2021.03.16.435261 Pages: 2021.03.16.435261 Section: New Results.
• [8] Taras Kowaliw, Nicolas Bredeche, and René Doursat (Eds.). 2014. Growing Adaptive Machines: Combining Development and Learning in Artificial Neural Networks. Studies in Computational Intelligence, Vol. 557. Springer Berlin Heidelberg, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-55337-0
• [9] Robert Tjarko Lange. 2022. evosax: JAX-based Evolution Strategies. arXiv
• [10] Robert Tjarko Lange, Tom Schaul, Yutian Chen, Tom Zahavy, Valentin Dallibard, Chris Lu, Satinder Singh, and Sebastian Flennerhag. 2023. Discovering Evolution Strategies via Meta-Black-Box Optimization. arXiv
• [11] Kaitlin Maile, Emmanuel Rachelson, Hervé Luga, and Dennis George Wilson. 2022. When, where, and how to add new neurons to ANNs. In Proceedings of the First International Conference on Automated Machine Learning (Proceedings of Machine Learning Research, Vol. 188), Isabelle Guyon, Marius Lindauer, Mihaela van der Schaar, Frank Hutter, and Roman Garnett (Eds.). PMLR, 18/1–12. Project website
• [12] Alexander Mordvintsev, Ettore Randazzo, Eyvind Niklasson, and Michael Levin. 2020. Growing Neural Cellular Automata. Distill (2020). https://doi.org/10.23915/distill.00023 https://distill.pub/2020/growing-ca/
• [13] Elias Najarro, Shyam Sudhakaran, and Sebastian Risi. 2023. Towards Self-Assembling Artificial Neural Networks through Neural Developmental Programs. arXiv
• [14] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv
• [15] Dan H. Sanes, Thomas A. Reh, and William A. Harris. 2007. Development of the nervous system (2. ed., [nachdr.] ed.). Elsevier, Academic Press, Amsterdam Heidelberg.
• [16] Samuel Schmidgall, Jascha Achterberg, Thomas Miconi, Louis Kirsch, Rojin Ziaei, S. Pardis Hajiseyedrazi, and Jason Eshraghian. 2023. Brain-inspired learning in artificial neural networks: a review. arXiv
• [17] Mari Sepp, Kevin Leiss, Florent Murat, Konstantin Okonechnikov, Piyush Joshi, Evgeny Leushkin, Lisa Spänig, Noe Mbengue, Céline Schneider, Julia Schmidt, Nils Trost, Maria Schauer, Philipp Khaitovich, Steven Lisgo, Miklós Palkovits, Peter Giere, Lena M. Kutscher, Simon Anders, Margarida Cardoso-Moreira, Ioannis Sarropoulos, Stefan M. Pfister, and Henrik Kaessmann. 2024. Cellular development and evolution of the mammalian cerebellum. Nature 625, 7996 (Jan. 2024), 788–796. https://doi.org/10.1038/s41586-023-06884-x Number: 7996 Publisher: Nature Publishing Group.
• [18] G. L. Shaw. 1986. Donald Hebb: The Organization of Behavior. In Brain Theory, Günther Palm and Ad Aertsen (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 231–233.
• [19] Kenneth O. Stanley, David B. D’Ambrosio, and Jason Gauci. 2009. A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks. Artificial Life 15, 2 (2009), 185–212. https://doi.org/10.1162/artl.2009.15.2.15202
• [20] Kenneth O. Stanley and Risto Miikkulainen. 2002. Evolving Neural Networks through Augmenting Topologies. Evolutionary Computation 10, 2 (June 2002), 99–127. https://doi.org/10.1162/106365602320169811
• [21] Kenneth O. Stanley and Risto Miikkulainen. 2003. A Taxonomy for Artificial Embryogeny. Artificial Life 9, 2 (April 2003), 93–130. https://doi.org/10.1162/106454603322221487
• [22] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. arXiv
• [23] Neele Wolterhoff and P. Robin Hiesinger. 2024. Synaptic promiscuity in brain development. Current biology: CB 34, 3 (Feb. 2024), R102–R116. https://doi.org/10.1016/j.cub.2023.12.037
• [24] Anthony M. Zador. 2019. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature Communications 10, 1 (Aug. 2019), 3770. https://doi.org/10.1038/s41467-019-11786-6 Number: 1 Publisher: Nature Publishing Group.