Optimal Policy Is Weakest Policy

Michael Timothy Bennett${}^1$ [0000−0001−6895−8782]
The Australian National University
michael.bennett@anu.edu.au

Abstract. Pancomputational Enactivism is a formalism of embodied, embedded, extended, and enactive intelligence. Previous work used this formalism to show the optimal choice of policy is the weakest. Experimental results support this claim. This has wide ranging implications. However there are flaws in its formal presentation, which undermine the optimality claims. Here we discuss these flaws, and present alternative proofs to rectify them.

Keywords: Pancomputational Enactivism, Bennett’s razor, w-maxing.

1 Introduction

Artificial intelligence is often framed as the pursuit of intelligent software. However the behaviour of software is determined by the hardware on which it runs. This distinction between software and hardware, and by extension between an agent and the environment in which it exists, undermines any claim that can be made about the behaviour of a theorist intelligence [1, 2]. This problem is called Computational Dualism [3]. Pancomputational Enactivism is a formalisation of cognition which addresses this problem. It formalises goal directed behaviour as embodied tasks, instead of as an interaction between a separate agent and environment. It is based upon Stack theory [4], which moves the “interpreter” from the level of the agent (for example a Turing machine in a body that interprets a software “mind”) to the physical laws of the environment¹. It avoids computational dualism by framing intelligent behaviour as a whole-of-system problem [5, 6, 3, 7]. It has subsequently been used to establish objective upper bounds on embodied intelligent behaviour [3, 8]. It has been used to show the optimal choice of policy is the weakest, and experimental results supported this claim [9–11]. This has wide ranging implications [12], from causality [13–15] to complex systems [8, 7, 16], to agents [17], to language and norms [18–20], the Fermi Paradox and the origins of life problem [21, 4], and above all consciousness [22–24, 4, 25, 26]. However, the early results contain a number of flaws. First, they use an early version of Pancomputational Enactivism [9] that differs substantially from that used in derivative works [3, 8, 7], which calls into question whether the proofs

still hold. Second, the proof that maximising the weakness of policies is sufficient to maximise the probability that they will generalise contains a miscount. This miscount does not change the result of the proof, but undermines its credibility. Finally, there is a flaw in the proof that it is necessary to maximise policy weakness to maximise the probability that policies will generalise. It failed to mention a dependency on the distribution of tasks. These problems do not refute the claims of optimality, but they certainly undermine them. Here we present alternative proofs of optimality, which do not suffer these flaws.

2 Definitions

We will begin with a summary of relevant definitions, followed by the proofs. These definitions are abridged versions of the full Pancomputational Enactivism formalism, but sufficient for the purposes of these proofs. Full length definitions are available in any one of the various full length published works on Pancomputational Enactivism [3].

1. Programs: $\Phi$is the set of states, and $P=2^{\Phi }$is all possible declarative programs. Each program is a potential point of difference between states². Assume a present state $\varphi \in \Phi$, and program $f\in P$is true if $\varphi \in f$.
2. Embodied Language: A body or abstraction layer is formalised as a finite vocabulary $\mathfrak{v}\subset P$. The embodied language is $L_{\mathfrak{v}}=\{l\subseteq \mathfrak{v}:\bigcap l\ne \varnothing \}$. Members of $L_{\mathfrak{v}}$are statements the body can express (state of memory etc). A statement $l\subseteq L_{\mathfrak{v}}$is true when $\varphi \in \bigcap l$. $E_l=\{y\in L_{\mathfrak{v}}:l\subseteq y\}$is called the extension of $l$.
3. $\mathfrak{v}$-Tasks: A $\mathfrak{v}$-task $\alpha =⟨I_{\alpha },O_{\alpha }⟩$has inputs $I_{\alpha }\subset L_{\mathfrak{v}}$and outputs $O_{\alpha }\subset E_{I_{\alpha }}$. Assume a uniform distribution over $\mathfrak{v}$-tasks. Let $\alpha$and $\omega$be $\mathfrak{v}$-tasks. If $I_{\alpha }\subset I_{\omega }$, $O_{\alpha }\subseteq O_{\omega }$, then $\alpha$ is examples of $\omega$. In previous work example tasks are referred to as children, and tasks they exemplify are referred to as parents. A $\mathfrak{v}$-task $\alpha$ is a child of $\mathfrak{v}$-task $\omega$ if $I_{\alpha }\subset I_{\omega }$ and $O_{\alpha }\subseteq O_{\omega }$. This is written as $\alpha ⊏\omega$. If $\alpha ⊏\omega$ then $\omega$ is then a parent of $\alpha$. $⊏$ implies a “lattice” or generational hierarchy of tasks. Formally, the level of a task $\alpha$ in this hierarchy is the largest $k$ such there is a sequence $⟨{\alpha }_0,{\alpha }_1,\dots ,{\alpha }_k⟩$ of $k$ tasks such that ${\alpha }_0=\alpha$ and ${\alpha }_i⊏{\alpha }_{i+1}$ for all $i\in (0,k)$.
4. Policies: $\pi \in L_{\mathfrak{v}}$is a correct policy for $\alpha$ if $E_{I_{\alpha }}\cap E_{\pi }=O_{\alpha }$. ${\Pi }_{\alpha }$ is the set of all correct policies for $\alpha$. The body learns or generalises to $\omega$ by inferring (choosing from ${\Pi }_{\alpha }$) a correct policy from examples $\alpha$ that is also correct for $\omega$, meaning $\pi \in {\Pi }_{\alpha }\cap {\Pi }_{\omega }$. ‘Experience’ is adding inputs and outputs to the examples $\alpha$. Intelligence is efficiency in learning³.
5. Heuristics: The weakness of a policy $\pi \in {\Pi }_{\alpha }$ is $|E_{\pi }|$. A proxy $<$ is a binary relation on statements, used to choose between policies implied by examples.

If $\pi$and ${\pi }^{\prime }$are correct policies for $\alpha$, $\alpha ⊏\omega$, and we are trying to choose one of $\pi$and ${\pi }^{\prime }$and maximise the chance of learning $\omega$from $\alpha$, then a proxy is used to choose between $\pi$and ${\pi }^{\prime }$. ${<}_w$is the weakness proxy. For statements $l_1,l_2$we have $l_1{<}_w{l}_2$iff $|E_{l_1}|<|E_{l_2}|$, meaning the proxy chooses $l_2$.

3 Proofs

Proposition 1 (sufficiency).
Assume $\alpha ⊏\omega$. The weakness proxy sufficient to maximise the probability that a parent $\omega$is learned from a child $\alpha$⁴.

Proof. You’re given the definition of $\mathfrak{v}$-task$\alpha$from which you infer a hypothesis $\pi \in {\Pi }_{\alpha }$. To learn $\omega$, you need $\pi \in {\Pi }_{\omega }$:

1. For every $\pi \in {\Pi }_{\alpha }$there exists a $\mathfrak{v}$-task${\gamma }_{\pi }\in {\Gamma }_{\mathfrak{v}}$s.t. $O_{{\gamma }_{\pi }}=E_{\pi }$, meaning $\pi$permits only correct outputs for that task regardless of input. We’ll call the highest level task ${\gamma }_{\pi }$s.t. $O_{{\gamma }_{\pi }}=E_{\pi }$the policy task of $\pi$.
2. $\omega$is either the policy task of a policy in ${\Pi }_{\alpha }$, or a child thereof⁵.
3. If a policy $\pi$is correct for a parent of $\omega$, then it is also correct for $\omega$. Hence we should choose $\pi$that has a policy task with the largest number of children. As tasks are uniformly distributed, that will maximise the probability that $\omega$is ${\gamma }_{\pi }$or a child thereof.
4. For the purpose of this proof, we say one task is equivalent⁶ to another if it has the same correct outputs.
5. No two policies in ${\Pi }_{\alpha }$have the same policy task⁷. This is because all the policies in ${\Pi }_{\alpha }$are derived from the same set inputs, $I_{\alpha }$.
6. The set of statements which might be outputs addressing inputs in $I_{\omega }$and not $I_{\alpha }$, is $E_{\overline{I_{\alpha }}}=\{l\in L_{\mathfrak{v}}:l\notin E_{I_{\alpha }}\}$⁸.
7. For any given $\pi \in {\Pi }_{\alpha }$, the extension $E_{\pi }$of $\pi$is the set of outputs $\pi$implies. The subset of $E_{\pi }$which fall outside the scope of what is required for the known task $\alpha$is $E_{\overline{I_{\alpha }}}\cap E_{\pi }$⁹.
8. $L_{\mathfrak{v}}=E_{I_{\alpha }}\cup E_{\overline{I_{\alpha }}}$and for all $\pi \in {\Pi }_{\alpha }$, $E_{\pi }\subset L_{\mathfrak{v}}$. Apart from the inputs and correct outputs of $\alpha$, $E_{\overline{I_{\alpha }}}$contains only outputs which would be incorrect according to both $\alpha$and $\omega$. Put another way, $E_{I_{\alpha }}\cap E_{\pi }=O_{\alpha }$for every possible choice of $\pi$in ${\Pi }_{\alpha }$. Hence the only way $|E_{\pi }|$can increase is if $|E_{\overline{I_{\alpha }}}\cap E_{\pi }|$increases. It follows that $|E_{\overline{I_{\alpha }}}\cap E_{\pi }|$increases with $|E_{\pi }|$.

9. $2^{|E_{I_{\alpha }}\cap E_{\pi }|}$is the number of non-equivalent parents of $\alpha$to which $\pi$generalises. It increases monotonically with the weakness of $\pi$.
10. Given $\mathfrak{v}$-tasksare uniformly distributed and ${\Pi }_{\alpha }\cap {\Pi }_{\omega }\ne \varnothing$, the probability that $\pi \in {\Pi }_{\alpha }$generalises to $\omega$is

$p(\pi \in {\Pi }_{\omega }\mid \pi \in {\Pi }_{\alpha },\alpha ⊏\omega )$is maximised when $|E_{\pi }|$is maximised. Recall from definition 4 that ${<}_w$is the weakness proxy. For statements $l_1,l_2$we have $l_1{<}_w{l}_2$iff $|E_{l_1}|<|E_{l_2}|$. $\pi$that maximises ${<}_w$will also maximise $p(\pi \in {\Pi }_{\omega }\mid \pi \in {\Pi }_{\alpha },\alpha ⊏\omega )$. Hence the weakness proxy maximises the probability that¹⁰ a parent $\omega$is learned from a child $\alpha$. $\square$

Proposition 2 (necessity).
To maximise the probability of learning $\omega$from $\alpha$, it is necessary to use weakness as a proxy.

Proof. Let $\alpha$and $\omega$be defined exactly as they were in the proof of prop. 1.

1. If $\pi \in {\Pi }_{\alpha }$and $E_{I_{\omega }}\cap E_{\pi }=O_{\omega }$, then it must be the case that $O_{\omega }\subseteq E_{\pi }$.
2. If $|E_{\pi }|<|O_{\omega }|$then generalisation cannot occur, because that would mean that $O_{\omega }⊈E_{\pi }$.
3. Therefore generalisation is only possible if $|E_{\pi }|\ge |O_{\omega }|$, meaning a sufficiently weak hypothesis is necessary to generalise from child to parent.
4. For any two hypotheses ${\pi }_1$and ${\pi }_2$, if $|E_{{\pi }_1}|<|E_{{\pi }_2}|$then the probability $p(|E_{{\pi }_1}|\ge |O_{\omega }|)<p(|E_{{\pi }_2}|\ge |O_{\omega }|)$because tasks are uniformly distributed.
5. Hence the probability that $|E_m|\ge |O_{\omega }|$is maximised when $|E_m|$is maximised. To maximise the probability of learning $\omega$from $\alpha$, it is necessary to select the weakest hypothesis.

To select the weakest hypothesis, it is necessary to use the weakness proxy. $\square$

4 Conclusion

In conclusion these updated proofs rectify the flaw in the count of tasks, and the unspecified distribution. They also use the more recent formulation of Pan-computational Enactivism, and may inform future research based on that formalism.

References

1. Leike, J., Hutter, M.: Bad universal priors and notions of optimality. Proceedings of The 28th Conference on Learning Theory, in Proceedings of Machine Learning Research pp. 1244–1259 (2015)

2. Orseau, L., Ring, M.: Space-time embedded intelligence. In: Bach, J., Goertzel, B., Iklé, M. (eds.) Artificial General Intelligence. pp. 209–218. Springer Berlin Heidelberg, Berlin, Heidelberg (2012)

3. Bennett, M.T.: Computational dualism and objective superintelligence. In: Artificial General Intelligence. Springer Nature (2024)

4. Bennett, M.T.: How To Build Conscious Machines. Ph.D. thesis, School of Computing, The Australian National University (2025), github.com/ViscousLemming/Technical-Appendices

5. Thompson, E.: Mind in Life: Biology, Phenomenology, and the Sciences of Mind. Harvard University Press, Cambridge MA (2007)

6. Piccinini, G., Maley, C.: Computation in Physical Systems. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy. Stanford University, Stanford, Sum. 21 edn. (2021)

7. Bennett, M.T.: Are biological systems more intelligent than artificial intelligence? (2025), forthcoming

8. Bennett, M.T.: Is complexity an illusion? In: Artificial General Intelligence. Springer Nature (2024)

9. Bennett, M.T.: The optimal choice of hypothesis is the weakest, not the shortest. In: Artificial General Intelligence. Springer Nature (2023)

10. Bennett, M.T.: Computable Artificial General Intelligence. Preprint (2022)

11. Bennett, M.T.: A formal theory of optimal learning with experimental results. Proceedings of the Thirty-fourth International Joint Conference on Artificial Intelligence (2025)

12. Bennett, M.T.: What the f*ck is artificial general intelligence? Springer Nature (2025)

13. Pearl, J., Mackenzie, D.: The Book of Why: The New Science of Cause and Effect. Basic Books, Inc., New York, 1st edn. (2018)

14. Bennett, M.T., Maruyama, Y.: The artificial scientist: Logicist, emergentist, and universalist approaches to artificial general intelligence. In: Goertzel, B., Iklé, M., Potapov, A. (eds.) Artificial General Intelligence. pp. 45–54. Springer Nature, Cham (2022)

15. Bennett, M.T.: Emergent causality and the foundation of consciousness. In: Artificial General Intelligence. Springer Nature (2023)

16. Simmons, G.: Comment on is complexity an illusion? Artificial General Intelligence (2025)

17. Perrier, E., Bennett, M.T.: Position: Stop acting like language model agents are normal agents (2025), arXiv

18. Bennett, M.T.: Symbol emergence and the solutions to any task. In: Artificial General Intelligence. Springer Nature (2022)

19. Bennett, M.T., Maruyama, Y.: Philosophical specification of empathetic ethical artificial intelligence. IEEE Transactions on Cognitive and Developmental Systems 14(2), 292–300 (2022)

20. Bennett, M.T.: On the computation of meaning, language models and incomprehensible horrors. In: Artificial General Intelligence. Springer Nature (2023)

21. Bennett, M.T.: Compression, the fermi paradox and artificial super-intelligence. In: Artificial General Intelligence. pp. 41–44. Springer Nature (2022)

22. Seth, A., Bayne, T.: Theories of consciousness. Nature Reviews Neuroscience (2022)

23. Ciaunica, A., Shmeleva, E.V., Levin, M.: The brain is not mental! coupling neuronal and immune cellular processing in human organisms. Frontiers in Integrative Neuroscience (2023)

24. Bennett, M.T., Welsh, S., Ciaunica, A.: Why Is Anything Conscious? Preprint (2024)

25. Evers, K., Farisco, M., Chatila, R., Earp, B., Freire, I., Hamker, F., Nemeth, E., Verschure, P., Khamassi, M.: Preliminaries to artificial consciousness: A multidimensional heuristic approach. Physics of Life Reviews 52, 180–193 (2025). DOI, ScienceDirect

26. Fields, C., Albarracin, M., Friston, K., Kiefer, A., Ramstead, M.J., Safron, A.: How do inner screens enable imaginative experience? applying the free-energy principle directly to the study of conscious experience. Neuroscience of Consciousness (2025)

27. Derrida, J.: Writing and difference. U of Chicago P (1978)