Research
Cite this article: Sacco F, Sakthivadivel D, Levin M. 2026 Topological constraints on self-organization in locally interacting systems. Phil. Trans. R. Soc. A 384: 20250011. https://doi.org/10.1098/rsta.2025.0011

Received: 25 January 2025
Accepted: 8 August 2025

One contribution of 18 to a theme issue ‘World models in natural and artificial intelligence’.

Subject Areas:
statistical physics, complexity, artificial intelligence

Keywords:
phase transitions, potts model, self-organization, emergence, diverse intelligence

Author for correspondence:
Dalton Sakthivadivel
e-mail: dsakthivadivel@gc.cuny.edu

Francesco Sacco${}^1$, Dalton Sakthivadivel${}^2$ and Michael Levin${}^{1,3}$

${}^1$Tufts University, Allen Discovery Center, Medford, MA, USA
${}^2$Department of Mathematics, CUNY Graduate Center, New York, NY, USA
${}^3$Tufts Center for Regenerative and Developmental Biology, Tufts University, Medford, MA, USA

GRAY

All intelligence is collective intelligence, in the sense that it is made of parts that must align with respect to system-level goals. Understanding the dynamics that facilitate or limit navigation of problem spaces by aligned parts thus impacts many fields ranging across life sciences and engineering. To that end, consider a system on the vertices of a planar graph, with pairwise interactions prescribed by the edges of the graph. Such systems can sometimes exhibit long-range order, distinguishing one phase of macroscopic behaviour from another. In networks of interacting systems, we may view spontaneous ordering as a form of self-organization, modelling neural and basal forms of cognition. Here, we discuss necessary conditions on the topology of the graph for an ordered phase to exist, with an eye towards finding constraints on the ability of a system with local interactions to maintain an ordered target state. By studying the scaling of free energy under the formation of domain walls in three model systems—the Potts model, autoregressive models and hierarchical networks—we show how the combinatorics of interactions on a graph prevent or allow spontaneous ordering. As an application, we are able to analyse why multiscale systems like those prevalent in biology are capable of organizing into complex patterns, whereas rudimentary language models are challenged by long sequences of outputs.

This article is part of the theme issue ‘World models in natural and artificial intelligence’.

1. Introduction

Self-organization is a fascinating phenomenon observed across diverse systems in nature and technology. In biology, self-organization of complex structures and functions, from subcellular machinery to multicellular morphogenesis, requires alignment of parts to navigate a problem space towards specific adaptive ends [1–7]. Dynamics of this form can be viewed as a sort of autopoietic cognition, raising interesting questions about how self-organizing systems navigate through configuration space given some target morphology [8–19]. More recently, current approaches to machine learning have involved physics-inspired models such as the Hopfield network [20–22] and spin glasses [23] and energy-based [24] or diffusion models [25]. In these systems, in order to maintain a pattern out of equilibrium or produce sensible data over long time spans, there must exist a ‘condensed’ phase with long-range order. While some systems—like those prevalent in biology—demonstrate remarkable abilities to organize at large scales, other systems that also exhibit some degree of collective intelligence—such as natural language models—have much more limited capabilities [26–30]. Despite this, spin systems have also been used to model cellular morphogenesis [31–35], suggesting the difference goes beyond the substrate of intelligence considered. This article is motivated by the following question: what is the functional distinction between simple language models and multicellular organisms, and can generative AI harness that property to achieve long-range order? In particular, we are interested in placing explicit thermodynamic bounds on the probability of some example systems occupying such a phase, with applications to estimating their capability to self-organize.

The question of how probable ordered configurations are as the number of elements increases is equivalent to asking about the existence of a phase transition, where the ultimate energetic constraints on the existence of an ordered phase arise from the topology of the interactions between subunits of the system. A quick argument due initially to Landau–Lifshitz [36]Landau and Lifshitz, §149 shows why the one-dimensional Ising model, the prototypical spin system, has no ordered phase at any temperature; their study of scaling is the technique we will use in this article, so we will review it here. Recall that a thermodynamically favourable change in state is one which decreases the free energy

delta F equals delta E minus T delta S

Suppose a domain wall of perimeter $P$P forms at an arbitrary location in the chain. Using an effective Hamiltonian, calculating the free energy of the system with zero domain walls and the system with one domain wall shows that the change in free energy scales like $-T\log P$minus T log P, meaning that forming a domain wall is always thermodynamically favourable for sufficiently large $P$P. As a result, long chains are unstable under thermal fluctuations, and disorder always propagates through the chain. In the thermodynamic limit, there is no spontaneous magnetization. Ising (1925)Ising proves this by finding the partition function and calculating the mean magnetization analytically in his celebrated 1925 paper, but the advantage of the argument of Landau and Lifshitz is that it applies generically to phase transitions in many different sorts of one-dimensional system (though the details prove somewhat subtle) and uses a simple scaling argument. The argument to the contrary in dimension greater than one is famously due to Peierls [37], where it is shown that the change in internal energy has a factor of $P$P balancing the change in entropy. This is found by computing the number of spins an ‘island’ interacts with—which is highly dependent on the properties of the lattice.

The main results in this article concern when an ordered phase can exist in a self-organizing system based on its interactions. We will focus on the scaling relationship between energy and entropy as it is determined by combinatorial topology. We begin this section by presenting our argument for a large universality class of models and deriving the necessary conditions for the existence of an ordered phase in this class. In §3, we analyse the one-dimensional Potts chain as a simple model system, recovering the argument given by Landau–Lifshitz in a more general setting. §4 maps autoregressive models [38] onto this one-dimensional framework and proves their

Figure 1. A local Hamiltonian in dimension one. The Hamiltonian of this chain is a sum of windowed Hamiltonians of length $\omega =5$omega equals five. Pictured is the second window; the first begins at $s_{-1}$s sub minus one.

inability to maintain long-range order. A discussion of transformers follows in §5section five. In §6section six, we apply these results to multiscale systems to evaluate the possibility of order in that context, based on the hierarchies built into the topology of interaction. Finally, we discuss implications of these results and future directions in §§7 and 8sections seven and eight. We conclude that certain systems cannot remain ordered over long ranges for all time and suggest this no-go theorem as a source of fitness pressure for biological phenomena like stigmergy and embodiment. The key insight of our work will be that topology is the critical factor differentiating these systems. Namely, while cells in the human body can coordinate and organize over vast scales, forming coherent tissues and organs, language models struggle to maintain consistency beyond their limited context windows. This disparity stems directly from the underlying topology of interactions in these systems.

2. The main argument

Consider a $k$k-vertex graph with $k$-by-$k$k by k adjacency matrix $G$G and an $n$n-ary variable on each vertex. Denote each such variable as $s_i$s sub i, indexed by $i\in \{1,\dots ,k\}$i from one to k. The interactions between any set of spins will be given by the Hamiltonian $H$H.

Definition 1. A windowed Hamiltonian $H$H is a Hamiltonian in that spin–spin interactions are defined only within a finite window of interaction $\omega$omega.

Remark. Without loss of generality, we will assume that the lowest energy state of any windowed Hamiltonian is zero. We will moreover assume the existence of an upper bound to the highest energy of any possible window Hamiltonian, $E_i^{\text{max}}⩽E^{\text{max}}$E max sub i is less than or equal to E max.
We call a sum of such effective Hamiltonians a local Hamiltonian. In particular,

Definition 2. A local Hamiltonian is a Hamiltonian which can be decomposed into a sum of several window Hamiltonians $H={\sum }_u{H}_u$H equals the sum over u of H sub u, all of which have the same window length $\omega$omega.

See figure 1 for a depiction. For the purpose of intuition, one may notice a local Hamiltonian with window size equal to one is a discrete Markov field.

Our methodology will be given by the following prescription:

1. Begin with the system in one of the ordered configurations, e.g. one of the stored patterns
2. Create a domain wall
3. Estimate the energy gained by the system
4. Find the asymptotics of the free energy as the number of domain walls increases

Computing the change in $F$F by changing the number of domain walls requires detailed knowledge of the combinatorics of the interactions on the graph. Instead, we can use the structure imposed by the windows. Namely, we have regulated the length of interactions such that we need only count the number of windows containing a domain wall. This does away with the particularities of the lattice and so can be applied to systems on very generic graphs. We will explore this in the present section.

The most general way to consider the interaction between the elements of our system is if the interactions are represented by a graph Hamiltonian. For simplicity, we will assume that the

Figure 2. Illustration of a graph Hamiltonian. Left: the graph $G$G represents the adjacency matrix of an undirected graph with six vertices, where edges indicate connections between vertices $s_i$s sub i. Right: the graph $H$H represents the graph Hamiltonian, where the coupling strengths $e_i$e sub i have come from $\bar{H}$H bar.

edges of the graph are fixed in time; one can easily justify this by appealing to an adiabatic approximation where the birth or death of edges is unlikely for the timescale on which observations are made.

Definition 3. Let $G$G be the adjacency matrix of a graph on $k$k vertices and $\bar{H}$H bar a $k$-by-$k$ matrix. A graph Hamiltonian is the Hamiltonian of a weighted directed graph; namely, a Hamiltonian which can be written as

H equals H bar entry-wise product G

where $\odot$the symbol is entry-wise multiplication and $\bar{H}$H bar has as entries the coupling strengths of the system.

See figure 2. We will now give a recipe for computing the scaling behaviour of energy and entropy for a graph Hamiltonian system. For the sake of simplicity, we will work with square planar embeddings of graphs (grid-like lattices) in this article. In these cases, a domain wall and its perimeter can be easily defined (figure 3).

Let $H$H be a graph Hamiltonian and $P$P be the perimeter length of a domain wall. As the perimeter length increases, the number of possible configurations of domain barriers increases, increasing the entropy of the system $\Delta S$delta S. We say that the entropy gained scales as $f_S$ if

delta S is of order f S of P

We give a similar definition for energy scaling: as the perimeter length increases, the upper and lower bounds of the energy gained scale as $O(f_E^{\text{high}}(P))$ and $O(f_E^{\text{low}}(P))$order f E high of P and order f E low of P, respectively. If $f_E^{\text{high}}=f_E^{\text{low}}\equiv f_E$f E high equals f E low, defined as f E we say that the energy gained scales as $f_E$, that is, $\Delta E=O(f_E(P))$delta E is of order f E of P.

If we can estimate how the energy and entropy scale, we can verify the existence or non-existence of an ordered phase without knowing the exact formulae for $E(P)$ and $S(P)$E of P and S of P, only considering how they scale as $P\to \infty$P goes to infinity.

Proposition 1. If $O(f_S)=O(f_E)$ then there exists an ordered phase.

Proof. Suppose $O(f)=O(f_S)=O(f_E)$. Then the change in free energy is

Figure 3. Two different domains in a two-dimensional grid. The perimeter that separates the two domains is drawn in orange.

If we now take $T$T to zero, the change in free energy becomes increasingly positive. As such, the creation of a domain wall is unfavourable. ■

In this way, we can rule out details of the system that could complicate the combinatorics—for example, the number of stored patterns. Let

$\pi =(\dots ,s_{-1},s_0,s_1,\dots )$pi equals the sequence s sub i

denote a ground state of the system. One may think of this state as a stored pattern or morphogenetic configuration. When there is a ground state degeneracy—that is, multiple states with zero energy exist, for instance, if a Hopfield model has $m>1$m greater than one stored patterns—they will be enumerated as

We can show that the number of such patterns affects neither the scaling of the entropy nor energy.

Lemma 1. Let $H={\sum }_u{H}_u$H equal the sum over u of H sub u. If there exist two energies $E^{\text{max}},E^{\text{min}}$E max and E min which are the greatest and least non-zero energy levels of all the windowed Hamiltonians $H_u$H sub u, respectively, then at thermal equilibrium, the ability to converge to an ordered phase is independent of energy levels and window sizes.

Proof. For any $H_u$H sub u let ${\omega }_1$omega one be the size of the smallest window and ${\omega }_2$omega two that of the largest. The energy gained from the creation of a domain wall is bounded by

In both cases, we have $E=O(P)$E is of order P so that the asymptotics of the system depend only on the perimeter length. ■

Lemma 2. Let $H$H be a graph Hamiltonian with $m>1$m greater than one stored patterns. At thermal equilibrium, the ability to converge to an ordered phase is independent of $m$m.

Proof. Let $B(P)$B of P be the number of possible configurations of domain barriers with perimeter $P$P.
The change in entropy due to the creation of a domain barrier can always be written as

delta S equals the log of m minus one times B of P, which is the log of B of P plus the log of m minus one.

In the thermodynamic limit, the term proportional to the number of barriers increases, while the term proportional to the number of patterns stored stays constant. As such, its contribution can be ignored. ■

Since neither the shape nor number of stored patterns affects the thermodynamics of the problem, and neither does the size of the interaction window, we reduce our analysis to the nearest-neighbour Ising model. Our main technical result is the following topological equivalence theorem:

Theorem 1. All local Hamiltonians on lattices with the same combinatorial structure have asymptotically equivalent free energies.

Proof. We will show that we can approximate any $H$H with the Hamiltonian of some arbitrary other model in such a way that their asymptotics are equivalent, implying that in the thermodynamic limit, the Peierls argument depends only on the perimeter length in the approximate system. Let $H={\sum }_u{H}_u$H equals the sum over u of H sub u be a local Hamiltonian on a planar graph $G$G with any spin $i$i coupled to ${\eta }_i$eta sub i other spins. Let $H_0$H zero be the Hamiltonian of an arbitrary system. One knows by the Bogoliubov inequality [39] that

F is less than or equal to F zero plus the expectation of H minus H zero with respect to the H zero ensemble

and therefore that if $H$H and $H_0$H zero are asymptotically equivalent then $F\sim F_0$F is asymptotic to F zero. As the perimeter length increases, we have $E=O(P)$E is order P and $E_0=O(P_0)$E zero is order P zero by Lemma 1. If the combinatorics of the lattices are the same—namely, if the set of ${\eta }_i$eta sub i is equal on both lattices—then we can take $P=P_0$P equals P zero. The change in the energy is computed by the Hamiltonian of the new configuration, implying $H-H_0\sim 0$H minus H zero is approximately zero. By Lemma 2, the probability of any disorder occurring is asymptotically the same in both systems, so that the change in free energies is also asymptotically equivalent. The claim follows. ■

Corollary 1. The capacity for self-organization in any system on a graph $G$G—that is, the existence or non-existence of a phase transition—is equivalent to that of a nearest-neighbour Ising model on the same graph, with

H zero equals minus J times the sum over i and j of s sub i times s sub j

and two stored patterns

pi one as a vector of ones and pi two as a vector of minus ones

An important remark is that we do not claim the phase transitions are the same. Naturally, they will differ in general, for instance, if there are more stored patterns in one system than the other. It is the existence of a phase transition from the topology of the lattice that we study.

3. Stored patterns in the windowed

With this in hand, we can study our first model system: the one-dimensional Potts model. The Potts model is a variant of the Ising model whose state vectors are richer than simply binary numbers. In particular, one can encode data in a Potts chain by choosing different spin configurations,

recalling the discussion in the Introduction. It is a suitable model to describe stored images or other configurations of multiply-valued variables.

Definition 4. Let $n$n be a finite positive number. A Potts chain $\mathcal{C}$C is a $\mathbb{Z}$-indexed set of $n$-ary integer symbols; that is, a chain of spins $s_i$s sub i indexed by $i\in \mathbb{Z}$i in the set of integers, where each $s_i$s sub i can assume any integer value in $\{1,\dots ,n\}$.

It follows from Theorem 1 that it is sufficient to consider the scaling of $\mathcal{C}$C to establish the existence of an ordered phase. In the same fashion as Landau–Lifshitz, it can be shown that local Hamiltonians in dimension one do not converge to a prescribed ordered state, since the formation of a domain wall always ‘interrupt’ the pattern.

Theorem 2. Let $H$H be a one-dimensional local Hamiltonian with $m>1$m greater than one stored patterns. At non-zero temperature, the formation of a domain wall is thermodynamically favourable.

Proof. Suppose that our Potts chain starts out in the first ground state or pattern, $\mathcal{C}={\pi }^1$C equals pi one. If

the change in free energy, delta F, equals delta E minus T delta S, which is less than zero

so that the free energy of the system decreases upon the formation of a domain barrier, then the formation of a domain barrier is thermodynamically favourable. It is immediate that any ${\pi }^{\alpha }$pi alpha differs from some other ${\pi }^{\gamma }$pi gamma by at least one change in spin. In a sequence of length $L$L there are $L-1$L minus one possible places where a domain wall can appear, and at each such place, we may obtain one of the $m-1$m minus one other patterns saved. As such, the change in entropy of the system is

delta S equals the log of, m minus one times L minus one

Upon the formation of a domain barrier, the windowed Hamiltonians that intersect it will have non-zero, positive energy. By assumption, the energy is bounded, and no more than $\omega$omega windows can be affected by a domain wall, from which we obtain

zero is less than or equal to delta E, which is less than or equal to omega times E max

The change in free energy is

delta F is less than or equal to omega E max minus T log of, m minus one times L minus one

Assume $T>0$T is greater than zero. As $L$L increases, the right-hand side of the equation eventually becomes negative. ■

4. Autoregressive models

We begin this section by recalling in figure 4 the motivation stated in the introduction: contrasting the generation of textual features with the generation of morphological features, we would like to understand how the self-organizing nature of text is constrained by the topology of the interactions between subunits generating that text.

We will now discuss a similar result as the one in the previous subsection, for autoregressive models. An autoregressive model is formed when values of a sequence are regressed against previous values of that sequence. Here, it will be defined as an estimator of some conditional probability distribution; namely, that of a sequence of observations of some data at step $i$i, given a history of observations until $i-1$i minus one.

Definition 5. Let $\{s_1,\dots ,s_{i-1}\}$the set s one through s i minus one be a random $n$-ary sequence of length $i-1$i minus one. Given a window (also called context) of length $\omega$omega, an order $\omega$omega autoregressive model computes

the probability of s i given the sequence from s i minus one to s i minus omega

as a probability vector over $\{1,\dots ,n\}$ by generating samples of $s_i$s i according to some random process on window states.

Such a function is called an AR($\omega$omega) model in particular. As shorthand, we will denote the estimator whose output is $v$v when given a sequence $\{s_{i-1},\dots ,s_{i-\omega }\}$s sub i minus one through s sub i minus omega as $M$M:

The function $M$M has the type of a conditional probability distribution. We further write

for the $c$-th component of $v$v. Conceptually, we have the picture in figure 5.

To fit this in the discussion of patterns and long-range order in spin chains, we find the following theorem useful. We will set the convention that if $u-\omega <1$u minus omega is less than one then the window is empty at that index, and that conditioning on the empty set is the same as taking unconditional probability.

Theorem 3. A unique local Hamiltonian with window length $\omega$omega can be associated to any AR($\omega$omega) model.

Proof. Let $M$M be our autoregressive model, and $s_{i-1},\dots ,s_{i-\omega }$s sub i minus one through s sub i minus omega our input sequence. The probability that the next observed spin in the sequence is equal to $c$c is

Suppose¹ we assign an energy to each possible $c$c with a scalar function $E:\{1,\dots ,n\}\to \mathbb{R}$E from the set one to n into the real numbers and constant weight $\beta ⩾0$beta greater than or equal to zero, such that

Set the constant in such a way that the lowest energy state has energy equal to zero. Now define a windowed Hamiltonian

for any $u$u in $\{1,\dots ,i\}$the set one to i so that the probability of any $u$th token occurring before $i$i is conditioned on the window preceding it. The full local Hamiltonian is the sum

in an obvious way. ■

Remark. By Theorem 3, the generation of samples by an autoregressive model can now be seen as sampling from a Boltzmann distribution over sequences.

We close this subsection with the following conclusion. The free energy can be calculated by $F=E-{\beta }^{-1}S$F equals E minus beta inverse S as usual. By Theorem 3, autoregressive models are one-dimensional systems described by a local Hamiltonian. The following corollary is then a consequence of the scaling argument in Theorem 2.

Corollary 2. For any finite $\beta$beta, an autoregressive model is unable to converge to a single stored pattern.

5. Transformers and attention

An interesting question to ask is what these results imply for the organizational capabilities, and hence intelligence, of large language models. Most state-of-the-art language models of today are

decoder-only transformer architectures (see [40]), meaning that to predict the next word (or token) they perform autoregression on all the previous elements of text—in other words, they are autoregressive at inference time. Indeed, transformers can be described as spin collectives [41–43], with training dynamics under asynchronous updates behaving like a spin chain evolving under Glauber dynamics and text generation being sampling from the corresponding equilibrium distribution, just as we have modelled autoregression by a spin chain here. A consequence of the results in §4 is a fundamental limitation on the coherence of a simple version of a large language model for long sequences of outputs—and hence an explanation for limitations observed in papers such as [30].

Proposition 2. Causally masked attention in a decoder-only model has no ordered phase.

Proof. Let $s\in {\mathbb{R}}^k$s in R to the k be the configuration of the network, $X$ a matrix of $m$ stored patterns $({\pi }^1,\dots ,{\pi }^m)$ with any $(X{)}_{\alpha }={\pi }^{\alpha }=(s_1,\dots ,s_k)$ and $X_{i\alpha }={\pi }_i^{\alpha }=s_i$, and $\beta$beta a constant scalar. Now $X$ is a $k$-by-$m$ matrix of $n$-ary variables. One knows (see [41]) the standard attention algorithm with the identity matrix for weights is a modern Hopfield network with an exponential potential function [22]. The asynchronous updates maximize the probability of a Gibbs distribution, where the energy $\beta E(s)$ of a configuration, $-\log P(s)+\text{const}$, is

With a causally-masked positional encoding from $i-\omega$ to $i$, the expression at $i$ with context length $\omega$omega becomes

This can easily be seen as an expression for a local Hamiltonian in a particular $\text{AR}(\omega )$A R omega model; it follows from Theorem 3 that we can apply Theorem 2 to the transformer, completing the proof of the statement. ■

For simplicity, we have used no mapping in an associative space; however, the result generalizes readily to those cases by the results in [41].

More sophisticated large language models employ multi-headed attention algorithms [44], where different attention heads in the decoder are attenuated to different classes of relationship. However, even in attention with heads for long-range dependencies, a maximum context length is imposed as a hyperparameter for reasons of computational cost (see the implementations in the review [45]). While the algorithm itself is more sophisticated and allows better performance when applied to two-dimensional data (e.g. images), the coherence of a sequence of text is constrained by context length, which is a realistic and necessary-to-consider constraint given hardware.

More discussion on this will be given at the end of §6.

6. Large hierarchical systems on graphs

In many systems of interest, there are hierarchical interactions, i.e. effective interactions between subgraphs. Many multiscale systems in biology and physics exhibit complex patterns consisting of pockets of regularity assembled into structures with global irregularity, such as tissues with different morphogenetic features assembled out of their constituent cells [46–51], functional networks in the brain consisting of regions specialized to process certain sorts of information [52–55], and self-assembling molecules in active matter situations [56–60]. In this section, we will investigate the interplay between the scaling of free energy at one level of a hierarchy and that of another,

Figure 4. Text generation in analogy to a morphogenetic process.

when the topology of the graph organizes those levels meaningfully, to quantify when such patterns are possible. We will complement the results in previous sections by showing that systems lacking hierarchical structure may be limited in their ability to form complex patterns.

Recall that a clique is a (non-empty) complete induced subgraph (figure 6). Suppose there exist $\ell >1$ell greater than one independent² cliques in $G$G, with $n_1,\dots ,n_{\ell }$n one through n ell vertices, respectively, and each $n_i>2$n i greater than two. There will be a macrostate associated with each clique giving the magnetization of that subgraph. We will argue that there are ways for local order but global disorder to exist in systems with cliques. Since the effective behaviour of the system, i.e. the properties of the cliques, is often a meaningful experimental variable (and hence a useful macrostate), this means organizing a system hierarchically can create interesting order phenomena. In particular, we want to show that there can exist ‘hierarchical behaviours’, where each clique individually is a coherent phase, but that phase varies from clique to clique.

For brevity, and without loss of generality³, we will assume the coupling constants $J$J are uniform across the graph. We will need the following observation before we prove our main theorem in this section. Recall that Boltzmann’s formula $S=k\log W$S equals k log W denotes by $W$W the multiplicity of a macrostate. A clique is positively (negatively) magnetized if all spins in the clique have value $+1$plus one ($-1$minus one). We will begin with $\ell$ell positively magnetized cliques. If the number (denote it $r$r) of ‘flipped’ (i.e. uniformly changed) cliques is a macrostate, then the multiplicity is the number of ways to arrange $r$r distinguished cliques out of the total $\ell$ell cliques. As such, we have

S equals k times the log of ell choose r

Theorem 4. Take any of the $\ell$ell cliques. There exist parameter regimes where individual cliques may change from positive to negative magnetization. For that temperature, take any individual $n_i$n sub i-clique and consider a spin within it. If the coupling of every spin in the clique is greater than $\frac{T}{2}\log n_i$ then the clique remains uniformly magnetized.

Proof. Flipping $r$r cliques from positive to negative magnetization takes an energy of ${\sum }_{\gamma =1}^{r}2J{n}_{i_{\gamma }}$, where $\gamma$gamma indexes cliques flipped, and has an entropy of the logarithm of $\ell$ell choose $r$r. The full change in free energy is

the change in free energy delta F equals the sum from gamma equals one to r of two J n sub i gamma, minus T log of the combination formula ell factorial over r factorial times ell minus r factorial

so that the scaling behaviour of $r$r flips to

where we have obtained the approximation of the entropy from $\left(\genfrac{}{}{0ex}{}{\ell }{r}\right)\approx {\ell }^r/r!$ and Stirling’s formula. Clearly, if (up to some constant)

then $\Delta F<0$delta F is less than zero. Hence, there exist temperatures for which it is favourable for some number $r$r of entire cliques to change. Fix such a $T$T. Within an $n_i$n sub i-clique, there are $n_i$n sub i ways to flip a single spin. This takes energy $2J$two J. If we have $2J>T\log n_i$, then there is no domain wall within the clique.

By minding constants in the $O(r)$ terms above, we show the hypothesis can be attained—namely, that there exist values of $T$T for which the inequality is satisfied.

Proposition 3. Let $n_{\text{max}}$n max be an integer greater than zero denoting the number of vertices in the largest clique. There exists a non-empty critical temperature range of hierarchical behaviour.

Proof. From the theorem above, we have

two J over log n sub i is greater than T, which is greater than two J times the sum of n sub i gamma, divided by r log ell minus r log r plus r

We also know that

r times n max is greater than or equal to the sum of n sub i gamma

Figure 5. An autoregressive model with $\omega =5$omega equals five.

and

so it suffices to take

It follows that

Since the upper bound is a strictly decreasing function of $n_{\max }$n max with a singularity at one, and the lower bound is a linear function of $n_{\max }$n max, they intersect for some sufficiently small slope, before which the inequality is satisfied. We conclude that there exist $\ell ,r$ell and r for which the set of satisfactory $T$T is non-empty. In particular, this occurs whenever

■

By looking at the effective behaviour of the graph, we consider the statistical properties of cliques to be like those of individual spins. When those cliques themselves form cliques, we have complete induced subgraphs on an effective graph and can make this argument again (see figure 7 for an example of what is meant).

This section demonstrates that whenever we have cliques at some level, there can be interesting hybridized behaviours where local order may exist, but the system may be globally disordered. This recapitulates observations of multiscale phenomena in biology and physics, where for some parameter regimes there is coherence at one level (e.g. the tissues constituting an individual organ) and non-uniformity at another (e.g. the differing organs in a body). It also shows that there can be order within a window of a transformer—if the attention heads have the right relationships—but is consistent with a lack of global coherence for all spins.

7. Applications and limitations

There are a number of compelling applications and important limitations of these results; we will discuss them here.

While a simplification, the fixed window models the practical context length limitations that real systems face. As the context increases, it becomes computationally infeasible to predict the next token—and only the last few words are fed as input to the model. Intuitively, this leads the model to completely forget what is left outside of its input window, placing constraints on how coherent the model is able to be over long periods of time. This will be a useful observation in §8. On the other hand, we have neglected to consider newly emerging strategies for expanding context or making the context window dynamic, such as those highlighted in [61]prior work. We hope to study these in the future.

Figure 6. A graph G with two independent three-vertex cliques and two edges connecting the cliques.

Theorem 1 is best applicable to systems falling under the Landau theory of equilibrium phase transitions. Many interesting non-equilibrium systems fall outside this regime, suggesting future directions which generalize these results to the kind of non-equilibrium free energies considered in stochastic thermodynamics [62–67]. However, in certain situations, the same scaling relationships are respected and simply become dynamic in time or localized in space [68,69], meaning the results generalize readily to those cases. At criticality and out of equilibrium, finite-size effects become relevant to the computations of critical exponents [39], but not to reasoning about the existence of phase transitions. Similarly, uniformity in coupling strength (as in Theorem 4) and window length (as in Theorem 3) can be assumed when studying asymptotics, since the only quantities of interest are the upper and lower bounds on the change in energy (as in Lemma 1).

8. Conclusions and future directions

In this article, we have developed a scaling argument extending Peierls’ argument, which can be used to reason about the possibility of spontaneous order in systems with local interactions.

Biological systems are known to exploit the complex, hierarchical structures formed by cells in multicellular organisms to enable self-organization and problem-solving in anatomical, physiological and gene expression spaces, as well as the three-dimensional space of conventional behaviour and linguistics [70,71]. Many recent discussions have focused on the similarities and differences in the dynamics that allow evolved, engineered and hybrid systems to bind their parts towards efficient navigation to large-scale goals—a capability which is ultimately a defining feature of ‘life’ [5,19,72–74].

In stark contrast, we have demonstrated that current natural language processing algorithms lack the necessary topology for self-organization. This limitation explains their inability to generate long and coherent text that matches the complexity and consistency of biological systems. The key insight of our results is that topology is the critical factor differentiating these systems. While cells in the human body can coordinate and organize over large scales, forming coherent tissues and organs, language models struggle to maintain consistency beyond their limited context windows. This disparity stems directly from the underlying topology of interactions in these systems.

By understanding the crucial role of topology in enabling self-organization, we can begin to envision new architectures for natural language processing and biologically inspired computing that might better emulate the self-organizing capabilities of biological systems. The inability of autoregressive large language models to maintain states of long-range order resembles the tangential speech or derailment in formal thought disorder, such as that found in schizophrenia and other forms of psychosis [75], lending a complementary aspect to the ‘hallucinations’ seen in current large language models. Proposed therapeutic approaches to formal thought disorder involve

(a) Full graph with all interactions

(b) Cliques and supercliques highlighted

(c) Effective graph obtained from supercliques

Figure 7. Consider a large (i.e. having many vertices) graph containing two or more complete induced subgraphs whose vertex sets are disjoint. An example is depicted in (a). In (b) the edges forming those independent cliques are highlighted in green. The edges between the consolidated cliques form two independent supercliques which are also highlighted in (b) in blue and red respectively, and edges participating in neither a clique nor a superclique are left black. Note that supercliques may consist of cliques with differing vertex number (e.g. the red superclique); also note that the supercliques shown in (c) form a further superclique with vertex set $\{\{1,2,3\},\{4,5,6\},\{7\}\}$consisting of groups one, two, and three; four, five, and six; and seven and edges coloured in black.

refining the articulated thought in conversation or written (i.e. diagrammatic) form under the guidance of a psychotherapist [76], suggesting that interacting with an environment is a way to enforce coherence of information.⁴ This further suggests that an embodied world model, extending the system in space and time by its interactions with an environment, can be leveraged to

Figure 8. Cognition as goal-directed behaviour, with a focus on producing and processing meaningful pieces of text.

maintain coherence. We hypothesize this explains why stigmergy and other forms of extracellular signalling arise in biological systems, which is known to enhance the ability for a collective system to order itself [77–82] and has been used in robot design and control for this feature [83–85]. Indeed, human brains are also unable to maintain the coherence of generated text past certain lengths of time due to energy constraints on neural processing, but have tools to cope with this, such as memory aids and cues in the environment.

Throughout, we have assumed that the free energy is minimized. For some systems, the equilibration time is sufficiently long that, in the interest of practicality, one would be interested in relaxing this assumption—for example, when the energy landscape is very rugged. One calls such systems spin-glasses [23].

Studying the phase transitions of glassy systems can be challenging even for relatively simple glasses, such as Hopfield networks [86,87] or the Edwards–Anderson model [88]. In forthcoming work, we will give similar estimates constraining the existence of phases with frozen disorder by the topology of the underlying lattice.

and editing; M.L.: conceptualization, funding acquisition, project administration, resources, supervision, writing—review and editing.

All authors gave final approval for publication and agreed to be held accountable for the work performed therein.

Authors’ Notes. An interactive preprint: [89] and video abstract: YouTube accompany this paper. We thank Karl Friston for helpful comments and Mikalai Shevko for assistance with code debugging.

Conflict of interest declaration. We declare we have no competing interests.

Funding. DARS is grateful for the support of the Einstein Chair programme at the CUNY Graduate Centre. ML gratefully acknowledges support of Karen Fries.

References

1. Couzin ID. 2007 Collective minds. Nature 445, 715–715. (doi:10.1038/445715a)

2. Watson RA, Buckley CL, Mills R. 2011 Optimization in ‘self-modeling’ complex adaptive systems. Complexity 16, 17–26. (doi:10.1002/cplx.20346)

3. Levin M. 2019 The computational boundary of a ‘self’: developmental bioelectricity drives multicellularity and scale-free cognition. Front. Psychol. 10, 2688. (doi:10.3389/fpsyg.2019.02688/full)

4. Kriegman S, Blackiston D, Levin M, Bongard J. 2020 A scalable pipeline for designing reconfigurable organisms. Proc. Natl Acad. Sci. USA 117, 1853–1859. (doi:10.1073/pnas.1910837117)

5. Watson RA, Levin M, Buckley CL. 2022 Design for an individual: connectionist approaches to the evolutionary transitions in individuality. Front. Ecol. Evol. 10, 823588. (doi:10.3389/fevo.2022.823588)

6. Baluška F, Miller WB, Reber AS. 2023 Cellular and evolutionary perspectives on organismal cognition: from unicellular to multicellular organisms. Biol. J. Linnean Soc. 139, 503–513. (doi:10.1093/biolinnean/blac005)

7. Miller WB, Baluška F, Reber AS. 2023 A revised central dogma for the 21st century: all biology is cognitive information processing. Prog. Biophys. Mol. Biol. 182, 34–48. (doi:10.1016/j.pbiomolbio.2023.05.005)

8. Stone JR. 1997 The spirit of D’arcy Thompson dwells in empirical morphospace. Math. Biosci. 142, 13–30. (doi:10.1016/s0025-5564(96)00186-1)

9. Deisboeck TS, Couzin ID. 2009 Collective behavior in cancer cell populations. Bioessays 31, 190–197. (doi:10.1002/bies.200800084)

10. Abzhanov A. 2017 The old and new faces of morphology: the legacy of D’Arcy Thompson’s ‘theory of transformations’ and ‘laws of growth’. Development 144, 4284–4297. (doi:10.1242/dev.137505)

11. Reber AS, Baluška F. 2021 Cognition in some surprising places. Biochem. Biophys. Res. Commun. 564, 150–157. (doi:10.1016/j.bbrc.2020.08.115)

12. Lyon P, Keijzer F, Arendt D, Levin M. 2021 Reframing cognition: getting down to biological basics. Phil. Trans. R. Soc. B 376, 20190750. (doi:10.1098/rstb.2019.0750)

13. Levin M. 2022 Technological approach to mind everywhere: an experimentally-grounded framework for understanding diverse bodies and minds. Front. Syst. Neurosci. 16, 768201. (doi:10.3389/fnsys.2022.768201)

14. Pio-Lopez L, Bischof J, LaPalme JV, Levin M. 2023 The scaling of goals from cellular to anatomical homeostasis: an evolutionary simulation, experiment and analysis. Interface Focus 13, 20220072. (doi:10.1098/rsfs.2022.0072)

15. Mathews J, Chang AJ, Devlin L, Levin M. 2023 Cellular signaling pathways as plastic, proto-cognitive systems: Implications for biomedicine. Patterns 4, 100737. (doi:10.1016/j.patter.2023.100737)

16. Levin M. 2023 Bioelectric networks: the cognitive glue enabling evolutionary scaling from physiology to mind. Anim. Cogn. 26, 1865–1891. (doi:10.1007/s10071-023-01780-3)

17. Levin M. 2023 Collective intelligence of morphogenesis as a teleonomic process. In Evolution ‘on purpose’: teleonomy in living systems. Cambridge, MA: The MIT Press. (doi:10.7551/mitpress/14642.003.0013)

18. Lagasse E, Levin M. 2023 Future medicine: from molecular pathways to the collective intelligence of the body. Trends Mol. Med. 29, 687–710. (doi:10.1016/j.molmed.2023.06.007)

19. Zhang T, Goldstein A, Levin M. 2025 Classical sorting algorithms as a model of morphogenesis: self-sorting arrays reveal unexpected competencies in a minimal model of basal intelligence. Adapt. Behav. 33, 25–54. (doi:10.1177/10597123241269740)

20. Hopfield JJ. 1982 Neural networks and physical systems with emergent collective computational abilities. Proc. Natl Acad. Sci. USA 79, 2554–2558. (doi:10.1073/pnas.79.8.2554)

21. Krotov D, Hopfield JJ. 2016 Dense associative memory for pattern recognition. Adv. Neural Inf. Process. Syst. 29.

22. Demircigil M, Heusel J, Löwe M, Upgang S, Vermet F. 2017 On a model of associative memory with huge storage capacity. J. Stat. Phys. 168, 288–299. (doi:10.1007/s10955-017-1806-y)

23. Castellani T, Cavagna A. 2005 Spin-glass theory for pedestrians. J. Stat. Mech.: Theory Exp. 2005, 05012. (doi:10.1088/1742-5468/2005/05/P05012)

24. Ranzato M, Poultney C, Chopra S, Le Cun Y. 2006 Efficient learning of sparse representations with an energy-based model. Adv. Neural Inf. Process. Syst. 19. (doi:10.7551/mitpress/7503.003.0147)

25. Yang L, Zhang Z, Song Y, Hong S, Xu R, Zhao Y, Zhang W, Cui B, Yang MH. 2023 Diffusion models: a comprehensive survey of methods and applications. ACM Comput. Surv 56, 1–39. (doi:10.1145/3626235)

26. Khandelwal U, He H, Qi P, Jurafsky D. 2018 Sharp nearby, fuzzy far away: how neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 284–294. Stroudsburg, PA, USA: Association for Computational Linguistics. (doi:10.18653/v1/P18-1027)

27. Sun S, Krishna K, Mattarella-Micke A, Iyyer M. 2021 Do long-range language models actually use long-range context? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 807–822. Stroudsburg, PA, USA: Association for Computational Linguistics. (doi:10.18653/v1/2021.emnlp-main.62)

28. Zhao Z, Wallace E, Feng S, Klein D, Singh S. 2021 Calibrate before use: improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, vol. 139, pp. 12697–12706, Proceedings of Machine Learning Research.

29. Malkin N, Wang Z, Jojic N. 2022 Coherence boosting: when your pretrained language model is not paying enough attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 8214–8236. Stroudsburg, PA, USA: Association for Computational Linguistics. (doi:10.18653/v1/2022.acl-long.565)

30. Kwa T et al. 2026 Measuring AI Ability to Complete Long Software Tasks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.

31. Graner F, Glazier JA. 1992 Simulation of biological cell sorting using a two-dimensional extended Potts model. Phys. Rev. Lett. 69, 2013–2016. (doi:10.1103/PhysRevLett.69.2013)

32. Chaturvedi R et al. 2005 On multiscale approaches to three-dimensional modelling of morphogenesis. J. R. Soc. Interface 2, 237–253. (doi:10.1098/rsif.2005.0033)

33. Torquato S. 2011 Toward an Ising model of cancer and beyond. Phys. Biol. 8, 015017. (doi:10.1088/1478-3975/8/1/015017)

34. Szabó A, Merks RMH. 2013 Cellular potts modeling of tumor growth, tumor invasion, and tumor evolution. Front. Oncol. 3, 87. (doi:10.3389/fonc.2013.00087)

35. Weber M, Buceta J. 2016 The cellular Ising model: a framework for phase transitions in multicellular environments. J. R. Soc. Interface 13, 20151092. (doi:10.1098/rsif.2015.1092)

36. Landau LD, Lifshitz EM. 1958 Statistical physics. vol. 5. Oxford, UK: Pergamon Press.

37. Peierls RE. 1936 On Ising’s model of ferromagnetism. Math. Proc. Camb. Philos. Soc. 32, 477–481. (doi:10.1017/S0305004100019174)

38. Box GEP, Jenkins GM. 1970 Time series analysis: forecasting and control. San Francisco, CA: Holden-Day.

39. Sakthivadivel DAR. 2022 Magnetisation and mean field theory in the Ising model. SciPost Phys. Lect. Notes 35. (doi:10.21468/SciPostPhysLectNotes.35)

40. Lin T, Wang Y, Liu X, Qiu X. 2022 A survey of transformers. AI Open 3, 111–132. (doi:10.1016/j.aiopen.2022.10.001)

41. Ramsauer H et al. 2021 Hopfield networks is all you need. In International conference on learning representations.

66. Lucarini V, Pavliotis GA, Zagli N. 2020 Response theory and phase transitions for the thermodynamic limit of interacting identical systems. Proc. R. Soc. A. 476, 20200688. doi:10.1098/rspa.2020.0688

67. Zakine R, Vanden-Eijnden E. 2023 Minimum-action method for nonequilibrium phase transitions. Phys. Rev. X 13, 041044. doi:10.1103/PhysRevX.13.041044

68. Hohenberg PC, Halperin BI. 1977 Theory of dynamic critical phenomena. Rev. Mod. Phys. 49, 435–479. doi:10.1103/RevModPhys.49.435

69. Marro J, Dickman R. 2005 Nonequilibrium phase transitions in lattice models. Cambridge, UK: Cambridge University Press.

70. Fields C, Levin M. 2022 Competency in navigating arbitrary spaces as an invariant for analyzing cognition in diverse embodiments. Entropy 24, 819. doi:10.3390/e24060819

71. Levin M. 2023 Darwin’s agential materials: evolutionary implications of multiscale competency in developmental biology. Cell. Mol. Life Sci. 80, 142. doi:10.1007/s00018-023-04790-z

72. McShea DW. 2013 Machine wanting. Stud. Hist. Phil. Sci. Part C: Stud. Hist. Phil. Biol. Biomed. Sci. 44, 679–687. doi:10.1016/j.shpsc.2013.05.015

73. Bongard J, Levin M. 2021 Living things are not (20th Century) machines: updating mechanism metaphors in light of the modern science of machine behavior. Front. Ecol. Evol. 9, 650726. doi:10.3389/fevo.2021.650726

74. Barwich AS, Rodriguez MJ. 2024 Rage against the what? The machine metaphor in biology. Biol. Phil. 39, 14. doi:10.1007/s10539-024-09950-4

75. Kircher T, Bröhl H, Meier F, Engelen J. 2018 Formal thought disorders: from phenomenology to neurobiology. Lancet Psychiatry 5, 515–526. doi:10.1016/S2215-0366(18)30059-2

76. Palmier-Claus J et al. 2017 Cognitive behavioural therapy for thought disorder in psychosis. Psychosis 9, 347–357. doi:10.1080/17522439.2017.1363276

77. Theraulaz G, Bonabeau E. 1999 A brief history of stigmergy. Artif. Life 5, 97–116. doi:10.1162/106454699568700

78. Marsh L, Onof C. 2008 Stigmergic epistemology, stigmergic cognition. Cogn. Syst. Res. 9, 136–149. doi:10.1016/j.cogsys.2007.06.009

79. Giuggioli L, Potts JR, Rubenstein DI, Levin SA. 2013 Stigmergy, collective actions, and animal social spacing. Proc. Natl Acad. Sci. USA 110, 16904–16909. doi:10.1073/pnas.1307071110

80. Gloag ES, Turnbull L, Whitchurch CB. 2015 Bacterial stigmergy: an organising principle of multicellular collective behaviours of bacteria. Scientifica 2015, 387342. doi:10.1155/2015/387342

81. Heylighen F. 2016 Stigmergy as a universal coordination mechanism I: definition and components. Cogn. Syst. Res. 38, 4–13. doi:10.1016/j.cogsys.2015.12.002

82. Sims R, Yilmaz Ö. 2023 Stigmergic coordination and minimal cognition in plants. Adapt. Behav. 31, 265–280. doi:10.1177/10597123221150817

83. Holland O, Melhuish C. 1999 Stigmergy, self-organization, and sorting in collective robotics. Artif. Life 5, 173–202. doi:10.1162/106454699568737

84. Valckenaers P, Germain BS, Verstraete P, Van Brussel H. 2007 MAS coordination and control based on stigmergy. Comput. Ind. 58, 621–629. doi:10.1016/j.compind.2007.05.003

85. Boldini A, Civitella M, Porfiri M. 2024 Stigmergy: from mathematical modelling to control. R. Soc. Open Sci. 11, 240845. doi:10.1098/rsos.240845

86. Amit DJ, Gutfreund H, Sompolinsky H. 1987 Statistical mechanics of neural networks near saturation. Ann. Phys. 173, 30–67.

87. Mézard M. 2017 Mean-field message-passing equations in the hopfield model and its generalizations. Phys. Rev. E 95, 022117. doi:10.1103/PhysRevE.95.022117

88. Edwards SF, Anderson PW. 1975 Theory of spin glasses. J. Phys. F: Met. Phys. 5, 965–974. doi:10.1088/0305-4608/5/5/017

89. Sacco F, Sakthivadivel D, Levin M. 2023 Requirements for self-organization (v1.1). Zenodo. doi:10.5281/zenodo.8416764