TTS

Computational Dualism and Objective Superintelligence

Michael Timothy Bennett${}^1$
${}^1$The Australian National University

November 12, 2024

Posted on 12 Nov 2024 — CC-BY 4.0 — [https://doi.org/10.36227/techrxiv.21965672.v7](https://doi.org/10.36227/techrxiv.21965672.v7) — This is a preprint. Version of Record available at [https://doi.org/10.1007/978-3-031-65572-2_3](https://doi.org/10.1007/978-3-031-65572-2_3)

Abstract

Awarded “Best Student Paper” at the 17th Conference on Artificial General Intelligence, 2024

The concept of intelligent software is flawed. The behaviour of software is determined by the hardware that “interprets” it. This undermines claims regarding the behaviour of theorised, software superintelligence. Here we characterise this problem as “computational dualism”, where instead of mental and physical substance, we have software and hardware. We argue that to make objective claims regarding performance we must avoid computational dualism. We propose a pancomputational alternative wherein every aspect of the environment is a relation between irreducible states. We formalise systems as behaviour (inputs and outputs), and cognition as embodied, embedded, extended and enactive. The result is cognition formalised as a part of the environment, rather than as a disembodied policy interacting with the environment through an interpreter. This allows us to make objective claims regarding intelligence, which we argue is the ability to “generalise”, identify causes and adapt. We then establish objective upper bounds for intelligent behaviour. This suggests AGI will be safer, but more limited, than theorised.

Computational Dualism and Objective Superintelligence

Michael Timothy Bennett${}^1$

[0000−0001−6895−8782]

The Australian National University
michael.bennett@anu.edu.au
http://www.michaeltimothybennett.com/

Abstract. The concept of intelligent software is flawed. The behaviour of software is determined by the hardware that “interprets” it. This undermines claims regarding the behaviour of theorised, software superintelligence. Here we characterise this problem as “computational dualism”, where instead of mental and physical substance, we have software and hardware. We argue that to make objective claims regarding performance we must avoid computational dualism. We propose a pancomputational alternative wherein every aspect of the environment is a relation between irreducible states. We formalise systems as behaviour (inputs and outputs), and cognition as embodied, embedded, extended and enactive. The result is cognition formalised as a part of the environment, rather than as a disembodied policy interacting with the environment through an interpreter. This allows us to make objective claims regarding intelligence, which we argue is the ability to “generalise”, identify causes and adapt. We then establish objective upper bounds for intelligent behaviour. This suggests AGI will be safer, but more limited, than theorised.

Keywords: enactivism · pancomputationalism · computational dualism · AGI · AI safety.

1 Introduction

AIXI [1]A I X I, as described in prior work, is a general reinforcement learning agent. It uses Solomonoff Induction, a formalism of Ockham’s Razor [2], to make accurate inferences from minimal data. It was initially thought to be pareto optimal, representing an upper bound on intelligence [3]. Unfortunately, this claim was later shown to be a matter of interpretation [4]. We argue that this is a valuable insight indicative of a much larger problem with how artificial intelligence (AI) is conceived. We call this problem “computational dualism”, in reference to René Descarte’s interactionist, substance dualism. We then discuss an alternative formulation of intelligent behaviour that permits objective performance claims. It is based upon enactive cognition [5], pancomputationalism [6] and weak constraint optimisation [7]. We use it to propose upper bounds on intelligent behaviour.

Intelligence: Our focus is not AIXI, but intelligence. General intelligence is often defined in terms of predictive models [1, 8, 9]. A more “intelligent” predictive model generalises, making accurate predictions in unfamiliar circumstances. It adapts. The more efficiently one adapts, the more “intelligent” one is. Predictive accuracy is not what matters. With enough training examples even a lookup table can make accurate predictions (because it will have seen every example). What matters is how efficiently one learns what caused examples. Knowing cause, one can make accurate predictions. Intuitively, a model explains the present by identifying those aspects of the past which caused it. Using such a model, we might use the more distant past to explain events in the more recent past, and the present to predict the future. It is the future with which we are concerned, as an agent that can accurately predict the results of its actions can choose the actions that yield the most reward. Hence the ability to adapt to unforeseen circumstances, satisfy goals and otherwise behave intelligently can be equated with the ability to identify cause and effect [10]. Of course, the only truyly accurate model of the environment is the environment. Everything else is an abstraction. Hence even if we only consider models that comprehensively explain the past, some will “generalise” better than others.

Simplicity: This is where Ockham’s Razor comes in. All else being equal, it implies that simpler models are more likely to generalise [11]. Simplicity is often formalised as Kolmogorov Complexity (KC) [12] or minimum description length [13]. The KC of an object is the length of the shortest self extracting archive of that object. Intuitively, the KC of the past is the shortest comprehensive description of the past. More compressed descriptions tend to generalise better. This is why some believe that compression and intelligence are closely related [14]. Formally, in the case of AIXI, if the model which generated past data is indeed computable, then the simplest model will dominate the Bayesian posterior as more and more data is observed. Eventually, AIXI will have identified the correct model, which it can use to generate the next sample (predict the future).

Subjectivity: We will now use AIXI to illustrate the problem with software “intelligence”. KC (and thus AIXI’s performance) is measured in the context of a UTM. By itself, changing the UTM would not meaningfully affect performance. When used in a universal prior to predict deterministic binary sequences, the number of incorrect predictions a model will make is bounded by a multiple of the KC of that model [2]. If the UTM is changed the number of errors only changes by a constant [15, pp. 2.1.1 & 3.1.1], so changing the UTM doesn’t change which model is considered most plausible. However, when AIXI employs this prior in an interactive setting, a problem occurs [4]. Intuitively (with significant abuse of notation), assume a program $f_1$f one is software, $f_2$f two is an interpreter and $f_3$f three is the reality (an environment, body etc) within which goals are pursued. According to Ockham’s Razor, AIXI is the optimal choice of $f_1$f one to maximise the performance of $f_3(f_2(f_1))$f three of f two of f one. However, in an interactive setting one’s perception of success may not match reality.

“Legg-Hutter intelligence [3] is measured with respect to a fixed UTM. AIXI is the most intelligent policy if it uses the same UTM.” [4, p.10]

Using our informal analogy of functions, this means performance in terms of $f_3(f_2(f_1))$f three of f two of f one depends upon $f_2(f_1)$f two of f one, not $f_1$f one alone. A claim regarding the performance of $f_1$f one alone would be subjective, in that it depends upon $f_2$f two.

“This undermines all existing optimality properties for AIXI.” [4, p.1]

Computational dualism: We suggest this problem has broader significance for AI. The concept of a software “mind” interacting with a hardware “body” echoes Descartes’ interactionist, substance dualism [16]. Descartes argued mental and physical “substances” interact through the pineal gland which interprets mental events to cause physical events [17], like a UTM interprets software. When later scholars pointed out the inconsistencies implied by this interaction between mental and physical, some argued that the mental “supervenes” on the physical, meaning any two objects that are exactly the same mentally must be exactly the same physically. More recently, philosophers have proposed purely physicalist depictions of the mind [18], and have even gone so far as to formalise cognition not just as embodied, but as enacted through the environment [5]. One might argue that AI is the engineering branch of philosophy of mind and cognitive science. Instead of mental and physical substances, we have software and hardware. Software “supervenes” on hardware; any two computers that are exactly alike in hardware configuration down to the value of each and every bit, must be exactly alike in terms of software. However, we have not moved on from dualism and formalised more recent conceptions of mind. AI still tends to be equated with “immortal” software. Unless there exists a Platonic realm of pure math (akin to a mental realm), Hinton’s (2023)Hinton’s “immortal” computations [19] do not exist and never have. There are only embodied “mortal” computations, because software is nothing more than the configuration of hardware. Our understanding of AI should be revised to reflect this. This is what we set out to do here.

Summary: We propose computational dualism, argue AI warrants an alternative, discuss one such alternative¹, and then propose an objective upper bound on intelligent behaviour.

2 Pancomputational Enactivism

To make objective claims we must avoid computational dualism. One alternative is enactivism [24]. It holds that mind, body and environment are inseparable. Cognition extends into the environment, and is enacted through what the organism does. For example, if someone uses pen and paper to solve a math problem, then cognition is enacted using the pen and paper [25]. To formalise enactivism, we must formalise cognition as a part of the environment. We do so in pancomputationalist [6] terms, albeit a very minimal interpretation thereof.

Pancomputationalism holds that everything is a computational system. Using the analogy from earlier, we formalise $f_2(f_3(f_1))$f two of f three of f one instead of $f_3(f_2(f_1))$f three of f two of f one. One may regard the interpreter $f_2$f two as the laws of nature, the environment $f_3$f three as software running on $f_2$f two, and we’re seeking to portray the mind $f_1$f one as subject to the environment. Because of this, the distinction between software and hardware must be discarded. Instead, we describe artificial minds in a purely behaviourist manner [9]. We describe inputs and outputs rather than a mechanism by which one is mapped to the other. Correct policies are then all possible “causal intermediaries” between a set of inputs and outputs, akin to functionalist explanations of human mentality [18].

The environment: Rather than assuming programs, we arrive at a pancomputational model of the environment by assuming first that some things exist, some do not, and that the environment is that which exists. Second, we assume there is at least one state of the environment. States could be differentiated along dimensions like time, but we don’t strictly need to assume that. We also don’t need to assume anything about the internal structure or contents of states. Instead, we can formalise the set of all declarative programs² as the powerset of states. A declarative program is “true” about every state it contains, and false about everything else. Every conceivable environment or state thereof amounts to a set of true declarative programs, or “facts”.

Definition 1 (environment).

• We assume a set $\Phi$phi whose elements we call states.
• A declarative program is $f\subseteq \Phi$f, a subset of phi, and we write $P$P for the set of all declarative programs (the powerset of $\Phi$phi).
• By a truth or fact about a state $\varphi$phi, we mean $f\in P$f in P such that $\varphi \in f$phi is an element of f.
• By an aspect of a state $\varphi$phi we mean a set $l$l of facts about $\varphi$phi s.t. $\varphi \in \bigcap l$phi is in the intersection of l. By an aspect of the environment we mean an aspect $l$l of any state, s.t. $\bigcap l\ne \varnothing$the intersection of l is not empty. We say an aspect of the environment is realised³ by state $\varphi$phi if it is an aspect of $\varphi$phi.

Our approach reflects Goertzel’s (2006)Goertzel’s framing of unary, dyadic and triadic relations [26]. A state here is unary, referring only to itself. However, a declarative program is dyadic relation between states, in the sense that it refers to states which are its truth conditions. Sets of declarative programs form a lattice based on truth conditions, which relates them to one another. We can then define triadic relations by taking three sets of declarative programs $i,o,\pi$i, o, and pi such that $\pi$pi is a “causal intermediary” implying $o$o given $i$i.

Embodiment: We use these dyadic relations to formalise enactivism [5], holding that everything we call the mind is just part of the environment, and “thinking” is just changes in environmental state. This blurs the line between agent and environment, making the distinction unclear. As Heidegger (1927)Heidegger maintained, Being is bound by context [27]. There is no need to define an agent that has no

environment, and so there seems to be little point in preserving the distinction. What we need is not a Cartesian model of disembodied intelligence looking in upon an environment but an embodiment, embedded in a particular part of the environment, through which goal directed behaviour can be enacted. We need to describe the circuitry with which cognition is embodied and enacted, much like a formal language only with truth conditions determined by whatever laws govern the environment. Just as every computational system we can build is finite, and living systems are ergodic [28], we assume all of this takes place within a system which has a finite number of configurations, and so amounts to a finite number of declarative programs. We call this an abstraction layer. Intuitively, an abstraction layer is like a window. It looks onto that part of the environment in which cognition takes place. For example, we might enumerate every possible declarative program which pertains only to one specific computer, and use that computer as our abstraction layer. However, as we can take any set of declarative programs and define an abstraction layer, it is “pancomputational” in the sense that it captures every system in computational terms.

Definition 2 (abstraction layer).

• We single out a subset $\mathfrak{v}\subseteq P$v contained in P which we call the vocabulary of an abstraction layer. The vocabulary is finite unless explicitly stated otherwise. If $\mathfrak{v}=P$v equals P, then we say that there is no abstraction.
• $L_{\mathfrak{v}}=\{l\subseteq \mathfrak{v}:\bigcap l\ne \varnothing \}$L sub v, the set of all subsets l of v such that the intersection of l is non empty is a set of aspects in $\mathfrak{v}$v. We call $L_{\mathfrak{v}}$L sub v a formal language, and $l\in L_{\mathfrak{v}}$l in L sub v a statement.
• We say a statement is true given a state iff it is an aspect realised by that state.
• A completion of a statement $x$ is a statement $y$ which is a superset of $x$. If $y$ is true, then $x$ is true.
• The extension of a statement $x\in L_{\mathfrak{v}}$ is $E_x=\{y\in L_{\mathfrak{v}}:x\subseteq y\}$. $E_x$E sub x is the set of all completions of $x$.
• The extension of a set of statements $X\subseteq L_{\mathfrak{v}}$ is $E_X={\bigcup }_{x\in X}{E}_x$.
• We say $x$ and $y$ are equivalent iff $E_x=E_y$.

(notation) $E$ with a subscript is the extension of the subscript⁴.

Intuitively, $L_{\mathfrak{v}}$L sub v is everything which can be realised in this abstraction layer. The extension $E_x$ of a statement $x$ is the set of all statements whose existence implies $x$, and so it is like a truth table. Now that we have an embodiment, we need to define goal directed behaviour. An abstraction layer is already goal directed in the sense that it constrains what might be described, just as living organisms evolve to thrive in particular environments. However, an organism can then engage in more specific goal directed behaviours by learning and adapting to remain fit in more circumstances. This is what we seek to formalise using an abstraction layer. It is where a predictive model might fit. However we do not need a value-neutral model of the environment [9].

“The best model of the world is the world itself.” - Rodney Brooks [29]

The only aspects of the environment that we might actually need model are those necessary to satisfy goals [30]. What we need is a model of a task, or more specifically to enumerate the goal directed behaviour (inputs and outputs) that will eventually cause the task to be completed. Goal directed here just means some outputs are “correct”, and some are not. It is arbitrary.

Definition 3 ($\upsilon$upsilon-task). For a chosen $\upsilon$upsilon, a task $\alpha$alpha is a pair $⟨I_{\alpha },O_{\alpha }⟩$I sub alpha, O sub alpha where:

• $I_{\alpha }\subset L_{\upsilon }$I sub alpha is a subset of L sub upsilon is a set whose elements we call inputs of $\alpha$alpha.
• $O_{\alpha }\subset E_{I_{\alpha }}$O sub alpha is a subset of E sub I sub alpha is a set whose elements we call correct outputs of $\alpha$alpha.

$I_{\alpha }$I sub alpha has the extension $E_{I_{\alpha }}$E sub I sub alpha we call outputs, and $O_{\alpha }$O sub alpha are outputs deemed correct. ${\Gamma }_{\upsilon }$Gamma sub upsilon is the set of all tasks given $\upsilon$upsilon.

(generational hierarchy) A $\upsilon$upsilon-task $\alpha$alpha is a child of $\upsilon$upsilon-task $\omega$omega if $I_{\alpha }\subset I_{\omega }$I sub alpha is a subset of I sub omega and $O_{\alpha }\subseteq O_{\omega }$O sub alpha is a subset of O sub omega. This is written as $\alpha ⊏\omega$alpha is a child of omega. If $\alpha ⊏\omega$alpha is a child of omega then $\omega$omega is then a parent of $\alpha$alpha. $⊏$The child relation implies a “lattice” or generational hierarchy of tasks. Formally, the level of a task $\alpha$alpha in this hierarchy is the largest $k$k such there is a sequence $⟨{\alpha }_0,{\alpha }_1,\dots {\alpha }_k⟩$alpha zero, alpha one, through alpha k of $k$ $\upsilon$upsilon-tasks where ${\alpha }_0=\alpha$ and ${\alpha }_i⊏{\alpha }_{i+1}$alpha zero equals alpha and alpha i is a child of alpha i plus one for all $i\in (0,k)$i in the range zero to k. A child is always “lower level” than its parents.

(notation) If $\omega \in {\Gamma }_{\upsilon }$omega is in Gamma sub upsilon, then we will use subscript $\omega$omega to signify parts of $\omega$omega, meaning one should assume $\omega =⟨I_{\omega },O_{\omega }⟩$omega equals I sub omega, O sub omega even if that isn’t written.

Intuitively, an input is a possibly incomplete description or task. An output is a completion of an input [def. 2]. We treat correctness as binary. An output is correct if it causes the task to become complete to some acceptable degree with some acceptable probability⁵. Degrees of complete or correct just reflect different $\upsilon$upsilon-task definitions⁶. For example we might put this in evolutionary terms, where the inputs and outputs are the enumeration of all “fit” behaviour in which an organism might engage, as remaining alive is goal directed behaviour. We now formalise learning and inference of goal directed behaviour as tasks. Inference requires a “policy”. Being a set of declarative programs, a correct policy is the goal of a $\upsilon$upsilon-task, so this amounts to goal learning.

Definition 4 (inference).

• A $\upsilon$upsilon-task policy is a statement $\pi \in L_{\upsilon }$pi in L sub upsilon. It constrains how we complete inputs.
• $\pi$pi is a correct policy iff the correct outputs $O_{\alpha }$O sub alpha of $\alpha$alpha are exactly the completions ${\pi }^{\prime }$pi prime of $\pi$pi such that ${\pi }^{\prime }$pi prime is also a completion of an input.
• The set of all correct policies for a task $\alpha$alpha is denoted ${\Pi }_{\alpha }$Pi sub alpha.⁷

Assume $\upsilon$upsilon-task $\omega$omega and a policy $\pi \in L_{\upsilon }$pi in L sub upsilon. Inference proceeds as follows:

1. we are presented with an input $i\in I_{\omega }$i in I sub omega, and

2. we must select an output $e\in E_i\cap E_{\pi }$e in the intersection of E i and E pi.
3. If $e\in O_{\omega }$e is in O omega, then $e$e is correct and the task “complete”. $\pi \in {\Pi }_{\omega }$pi in Pi omega implies $e\in O_{\omega }$e in O omega, but $e\in O_{\omega }$e in O omega doesn’t imply $\pi \in {\Pi }_{\omega }$pi in Pi omega (an incorrect policy can imply a correct output).

Intuitively, a policy constrains how we complete inputs. It is a correct policy if it constrains us to correct outputs. To “learn” a policy we use a proxy. A proxy estimates one thing, by measuring another seemingly unrelated thing. For example, AIXI uses simplicity to estimate model veracity. In our case, we want a policy that classifies correct outputs. We will use a proxy called “weakness”, which has been shown to outperform simplicity in sample efficiency and causal learning [7, 20]. Where simplicity is a property of form, weakness is a property of function. In order to make objective claims regarding performance, we cannot rely upon subjective interpretations of form.

Definition 5 (learning).

• A proxy $<$less than is a binary relation on statements. ${<}_w$less than w is the weakness proxy. For statements $l_1,l_2$l one and l two we have $l_1{<}_w{l}_2$ iff $|E_{l_1}|<|E_{l_2}|$l one is less than w l two if and only if the size of E l one is less than the size of E l two.

(generalisation) A statement $l$l generalises to a $\mathfrak{v}$-task $\alpha$v-task alpha iff $l\in {\Pi }_{\alpha }$l is in Pi alpha. We speak of learning $\omega$omega from $\alpha$alpha iff, given a proxy $<$less than, $\pi \in {\Pi }_{\alpha }$pi in Pi alpha maximises $<$the proxy relative to all other policies in ${\Pi }_{\alpha }$Pi alpha, and $\pi \in {\Pi }_{\omega }$pi in Pi omega.

(probability of generalisation) We assume a uniform distribution over ${\Gamma }_{\mathfrak{v}}$Gamma v. If $l_1$l one and $l_2$l two are policies, we say it is less probable that $l_1$l one generalizes than that $l_2$l two generalizes, written $l_1{<}_g{l}_2$l one is less than g l two, iff, when a task $\alpha$alpha is chosen at random from ${\Gamma }_{\mathfrak{v}}$Gamma v (using a uniform distribution) then the probability that $l_1$l one generalizes to $\alpha$alpha is less than the probability that $l_2$l two generalizes to $\alpha$alpha.

(sample efficiency) Suppose $\mathfrak{app}$app is the set of all pairs of policies. Assume a proxy $<$less than returns 1 iff true, else 0. Proxy ${<}_a$less than a is more sample efficient than ${<}_b$less than b iff

$$\left(\sum _{(l_1,l_2)\in \mathfrak{app}}|(l_1{<}_g{l}_2)-(l_1{<}_a{l}_2)|-|(l_1{<}_g{l}_2)-(l_1{<}_b{l}_2)|\right)<0$$

(optimal proxy) There is no proxy more sample efficient than ${<}_w$less than w, so we call ${<}_w$less than w optimal. This formalises the idea that “explanations should be no more specific than necessary” (see Bennett’s razor in [7]).

Learning is an activity undertaken by some manner of agent, and a task has been “learned” when that agent knows a correct policy. For example, one has “learned” chess when one knows the rules and some winning strategies. Here instead of agents, we have embodied goal-directed behaviours in the form of $\mathfrak{v}$-tasks. Humans typically learn from “examples”. In the context of a $\mathfrak{v}$-task an “example” is a correct output and an input that is a subset thereof. A collection of examples is a child task. “Learning” is an attempt to generalise from a known child to one of its parents. Intuitively, a child functions like a “history” or memory

of correct interactions (an “ostensive definition”). The lower the child is in the generational hierarchy one learns from, the more sample efficiently one learns. We assume tasks are uniformly distributed because anything else would imply that environment “prefers” or makes a value judgement about goals. The environment is a value judgement about what exists, but is otherwise assumed to be impartial. Consequently the most sample efficient proxy is ${<}_w$the weakness proxy [7]. It enables causal learning [20] as the weakest correct policies have the highest probability of being the causal intermediary between inputs and outputs.

3 Limits of Intelligence

As stated earlier, intelligence is often understood in terms of “the ability to generalise” [8] and learn the policy which “caused” examples [20]. We wish to understand the objective upper limits of intelligence. ${<}_w$The weakness proxy is the optimal choice of proxy for sample efficiency [7, prop. 1, 2, 3]. In the context of a fixed abstraction layer, the upper bound of intelligent behaviour is attained by the weakness proxy. However, more intelligence is not always useful. Intuitively, making a human more intelligent is unlikely to improve their driving. Likewise, intelligence conveys no advantage if one’s purpose is to ascribe meaning to random noise. The extent to which it is possible to construct weak correct policies depends on both the abstraction layer and the task. An abstraction layer is a bottleneck on intelligence, and can be goal directed just like a task. We can measure this goal directed “utility” as the extent to which it is possible to construct weaker correct policies given a task in a particular abstraction layer.

Definition 6 (utility of intelligence). Utility is the difference in weakness between the weakest and strongest correct policies of a task. The utility of a $\mathfrak{v}$-task $\gamma$v-task gamma is $ϵ(\gamma )={\max }_{\pi \in {\Pi }_{\gamma }}(|E_{\pi }|-|O_{\gamma }|)$epsilon of gamma equals the maximum over pi in capital pi sub gamma, of the size of the extension of pi minus the size of the task gamma.

To maximise probability of generalisation, we use the weakness proxy and try to maximise utility. Increasing utility changes the abstraction layer to allow construction of weaker correct policies. Beyond a certain point, utility may be increased by increasing $|L_{\mathfrak{v}}|$the size of L v without actually changing the weakest correct policies ${\Pi }_{\gamma }$capital pi sub gamma contains (just the size of their extensions). This means it is not always necessary to increase utility to construct policies that generalise, but it is helpful up to a point. Utility is maximised when $\mathfrak{v}=P$v equals P, though in practice finite resources would limit us to smaller vocabularies. ${\Gamma }_P$Gamma sub P contains all tasks in all vocabularies. Hence, for every task $\rho$rho in ${\Gamma }_P$Gamma sub P we can define a function that takes a vocabulary $\mathfrak{v}$v and returns a $\mathfrak{v}$-task which is a child of $\rho$rho. This lets us represent a task in different vocabularies to compare their utility.

Definition 7 (uninstantiated-tasks). The set of all tasks with no abstraction (meaning $\mathfrak{v}=P$v equals P) is ${\Gamma }_P$Gamma sub P (it contains every task in every vocabulary). For every $P$-task $\rho \in {\Gamma }_P$P-task rho in Gamma sub P there exists a function ${\lambda }_{\rho }:2^P\to {\Gamma }_P$lambda sub rho from the power set of P to Gamma sub P that takes a vocabulary ${\mathfrak{v}}^{\prime }\in 2^P$v prime in the power set of P and returns a ${\mathfrak{v}}^{\prime }$-task $\omega ⊏\rho$v prime task omega which is a child of rho. We call ${\lambda }_{\rho }$lambda sub rho an uninstantiated-task. It is instantiated by choosing a vocabulary.

Proposition 1 (upper bound). The most ‘intelligent’ choice of policy and vocabulary given uninstantiated task ${\lambda }_{\rho }$lambda rho is $\pi$pi and $\mathfrak{v}$v s.t. $\mathfrak{v}$ maximises utility for ${\lambda }_{\rho }(\mathfrak{v})$, $\pi \in {\Pi }_{{\lambda }_{\rho }(\mathfrak{v})}$pi in capital pi sub lambda rho of v and $\pi$ maximises weakness.

Proof: We have equated intelligence with sample efficient generalisation. According to [7, prop. 1, 2]prior work the weakest correct policies have the highest probability of generalising. Given an uninstantiated task ${\lambda }_{\rho }$lambda rho, utility measures the weakness of the weakest correct policies. We can use this to compare vocabularies. By choosing a vocabulary $\mathfrak{v}$v which maximises utility for ${\lambda }_{\rho }(\mathfrak{v})$, we instantiate ${\lambda }_{\rho }$ in a vocabulary that maximises the weakness of correct policies for ${\lambda }_{\rho }$. Then, using weakness proxy, we can select a policy that has the highest possible probability of generalising, and thus maximise sample efficiency. ■

Put another way, utility is maximised for ${\lambda }_{\rho }(\mathfrak{v})$ when $\mathfrak{v}=P$. When utility is maximised there must exist $\pi \in {\Pi }_{{\lambda }_{\rho }(\mathfrak{v})}$ which is a weakest policy in ${\Pi }_{{\lambda }_{\rho }(P)}$.

Concluding remarks: Here we argued software minds are a flawed concept, symptomatic of “computational dualism”. We argued for an alternative we call pancomputational enactivism, which allows for objective claims regarding behaviour. We used this alternative to propose upper bounds on intelligence, and we anticipate these results may further our understanding of AI safety and AGI. In practical terms this upper bound is unattainable, because we can only build systems with finite vocabularies. Moreover, too large a vocabulary $\mathfrak{v}$v could make $\mathfrak{v}$-tasks intractable. Physical embodiment severely constrains what is possible. Rigorous research has been undertaken regarding the risks of AGI [32], but it is based upon computational dualism. Our results suggest AGI will be safer, but more limited, than has been theorised. Our results may also be of use in the pursuit of AGI. Higher utility does mean weaker and more generalise-able policies, which suggests one should optimise for higher utility whilst trying to minimise $|\mathfrak{v}|$the size of v.

References

[references omitted]

[references omitted]

[references omitted]

Footnotes

1. Definitions or variations thereof are shared with related work [7, 20, 21, 22, 23]. ↩

2. Intuitively, declarative programs are anything which is true or false. ↩

3. Realised meaning it is made real, or brought into existence. ↩

4. e.g. $E_l$ is the extension of $l$. ↩

5. Also called “satisficing” a goal [31]. ↩

6. Affect or reward and attribution thereof are beyond this paper’s scope. ↩

7. To repeat the definition in set builder notation: ${\Pi }_{\alpha }=\{\pi \in L_{\upsilon }:E_{I_{\alpha }}\cap E_{\pi }=O_{\alpha }\}$ ↩