Jeff Clune keynote at ALIFE: open-endedness, AI-generating algorithms and AI scientists

Analysis · Report with the audio

Introduction of the speaker [0:01]

[0:01] Speaker A: Okay, so we are ready. So, uh, welcome back.

[0:05] So I’m very happy to, uh, introduce our next keynote speaker, um, Jeff Clune. Uh, Jeff is a, um, um, professor at the University of British Columbia, and, uh, more recently he has just founded a startup called Recursive, which amazingly raised quite a, a big amount of money in a very short time, so congratulations. I hope you will get, we’ll, uh, get some tips from you, but check it out. So, um, also, he has been previously, uh, working at Uber AI Labs, and, uh, he holds a PhD from, uh, the University of Michigan, and later on, a professor at the University of Wyoming. Yes, here we have the proud supervisor.

[0:50] Jeff Clune: Michigan State.

[0:51] Speaker A: Michigan S- oh. Sorry, sorry. Michigan State. And, uh, yeah, and then, uh, you also had been working for, um, uh, Google DeepMind as an advisor and, uh, OpenAI as a lead for open-endedness. And more remarkably, Jeff, uh, has been very active in the ALife community before. He’s actually one of the winners of the Early Career, uh, uh, Award from the ISAL, the International Society for Artificial Life. So it’s great to hear how ideas from… that originated in this community then, uh, became, uh, quite successful and mainstream even in the industry, and in particular ideas from open-endedness and how can this be used for improving AI algorithms. So Jeff will talk about, uh, AI scientists and, uh, open-ended and AI-generating algorithms.

[1:41] Speaker A: So the floor is your Jeff. Thanks for joining.

Roots in the ALife community [1:44]

[1:44] Jeff Clune: Thank you. [clapping] Are you ready or should I wait? All right. Well, thank you very much for the invitation. It’s really nice to be here and see so many old friends. Hello, Chris. Um, I start— wanted to start just by saying that I have a lot of deep roots. Are you gonna work? You can do it. No, you’re not gonna work. Uh, I have a lot of roots in the ALife community. Oh, this isn’t working now.

[2:25] Okay, that’ll work. Uh, I have a lot of roots in artificial life, so it’s, uh, holds a special place in my heart.

[2:30] Jeff Clune: In fact, my first ever academic paper was at Artificial Life in two thousand and four, ALife nine. My first ever talk was here at this conference, and this is— I actually went and grabbed a slide from the original talk. So for the younger ones in the audience, this is what life used to be like before you had image generation algorithms to make your slides. So, um, I did a PhD with Charles here, as well as Rod Pennock and Rich Lenski.

[2:55] Chris Adami, um, was amongst the, the crew. All of these wonderful people I worked with here, uh, early on with Avida, uh, and then with evolving neural networks, and I went on to be a member of the board of directors of ISAL. Uh, all of which is to just say that I have a deep connection to the ALife community, and, uh, it is true that lately I’ve been focusing on other areas of AI and going to other conferences, but in many ways, this feels kind of like my academic home, and so I just wanted to say it’s great to be home, uh, and thank you for the invitation.

The objective paradox and innovation engines [3:25]

[3:25] Jeff Clune: Oh, no, this is not working. That’s a bummer. Okay.

[3:31] If I could get a, a, a clicker that would work, that would be really great. All right, so it’s great to be home, so thank you.

[3:36] So I wanted to start off the academic part of the talk by pointing out that there’s a paradox in life, which is that if you try too hard to solve a problem… Now it’s working, so I’m gonna risk it. You will fail. If you try too hard and the problem is really, really hard, you will fail. In contrast, if you completely ignore the, ignore the objective and just go off and do interesting things in life, do basic science, you’re much more likely to succeed. And as a metaphor for that, imagine that you are an agent in this maze and you’re being incentivized to only get closer to the goal, which is here. Well, then this algorithm did that, and it goes north and bounces its head against the wall forever because the reward function is discouraging it from going anywhere away from north.

[4:20] Jeff Clune: In contrast, if you use novelty search, you just trivially go and explore this domain, you’re far more likely to succeed. Now, you might think that that is overly simple, that doesn’t show up in real life, but in truth, that is the story behind every major technological challenge that has been overcome by humanity. So for example, if you go back thousands of years to cooking over an open fire, and you said, “I will only fund scientists in my queendom that give me faster cooking with less smoke,” you would never invent the modern microwave, because to do that, you had to have been working on radar technology

[4:53] and notice that a chocolate bar melted in your pocket. Similarly, if you were working on— If you went back to the time of the abacus, and you only funded scientists who give you more and faster compute, you would never invent a computer, because to do that, you had to have been working on vacuum tubes and electricity, which were technologies that were not invented because they produced computation.

[5:16] Jeff Clune: They were not being rewarded by the goal. So the conjecture here is that you have to invent your own goals as you go, kind of catch chance on the wing, create new problems, notice new things to work on, and eventually, that’s how you unlock the hardest things in science.

[5:33] So that brings us to this idea of goal switching. If you are, for example, a roboticist and you’re trying to get a robot to do bipedal locomotion, and you’re measuring that by forward speed, and you start to see it doing other things, like balancing on one leg or crawling on all fours, you shouldn’t throw that out because it doesn’t measurably improve forward speed. You should embrace those new opportunities and those new challenges because they might be essential stepping stones that ultimately unlock bipedal locomotion. So the eco— we want our algorithms to basically catch serendipity, to notice, “Aha, that’s funny, that’s interesting.

[6:07] Jeff Clune: I should work on that. That’s now possible. Let me see if I can do that.” I, I even want them to do basic science.

[6:14] So I call these things innovation engines, and human culture and evolutionary biology are two examples, Darwinian evolution. They basically operate with the same principle, which is you start with a archive of the inventions you have so far, you take them out, you change them a little bit, maybe you recombine them, and if they’re either the, a better version of something you already have, a better mousetrap if you have a, um, a mousetrap, or they’re totally new, like an iPhone and you didn’t have that before, then add them to the set. And just repeat that and grow an increasingly large expanding set of interesting different artifacts.

Quality diversity: MAP-Elites and Go-Explore [6:53]

[6:53] Jeff Clune: So today, what I wanna do is m- basically motivate and introduce three different kinds of algorithms that are trying to do that, that are trying to be, to produce innovation engines. Uh, QD algorithms, open-ended algorithms, and AI generating algorithms.

[7:06] So quality diversity algorithms are, uh, their goal is not like typical optimization. They’re not trying to find the best Go player imaginable. They’re trying to find a large, diverse set of high-quality solutions. Just like nature produced all of these wildly different solutions to the problem of life, each of which is amazing at what it does.

[7:28] So the poster child of q- quality diversity has become the MAP-Elites algorithm, which was co-invented by Jean-Baptiste Mouret and I over a decade ago, and it’s quite simple, and I wanna walk you through this algorithm because it’s gonna be the same basic playbook that we’re gonna see in almost all of the work that I present today.

[7:43] Jeff Clune: So in MAP-Elites, you choose dimensions of behav- of, uh, variation you care about. For example, here, the height and the weight of robots. And then you say, uh, you basically run a pretty simple, uh, algorithm. So you start, you have a parameterized, uh, like a parameter vector that describes a robot. You evaluate it. You get its properties according to the dimensions of variation you care about. You have some performance, which I’m showing here in color, and you put it into its l- proper location in the map. And then you just run a loop forever, which is pull something out of the archive, change it a little bit, reevaluate it, figure out where it goes in the map, and if it’s the best thing you’ve ever seen at that point in space, swap it in.

[8:25] And I hope you can see that that is basically acting like an innovation engine.

[8:30] Jeff Clune: For one, rather than give you a single solution, it returns a surface or a set of solutions that teach you a lot about your fitness landscape, where there are local optima and minima.

[8:39] But also, it’s acting like the innovation engine. It’s got an archive of things found so far, pulls it out, changes it, adds it. If it’s new, you’re good. You don’t even have to compete. And if you’re, there’s something already there, you have to compete, you have to be higher quality. So you get diversity and quality.

[8:54] So, uh, when I was a postdoc in Hod Lipson’s lab, I was working on this soft robotics problem, and we had done this paper a year before, and it made a lot… People were really happy with this video ‘cause we got a lot of fun, diverse, quirky creatures. But there was a cheat, which is that every single creature in this video came from a different run of evolution. Within one run of evolution, there was almost no diversity at all, just tiny little variations on, on a theme.

[9:19] Jeff Clune: I’m sure you’re all very familiar with this. But that’s not what happens in nature. What we want is one run to explode in complexity and keep innovating forever.

[9:27] And so we tried classic optimization on this problem. It does terribly, mostly black dots. We tried multi-objective, um, Pareto front, trading off diversity and performance. Interestingly, you get higher performance, we know that diversity helps, but you actually get less exploration ‘cause now it’s got these really good local optima, and it really doesn’t wanna trade off performance in that diversity. So you get very little exploration of the s- of the search space. Now, with the exact same amount of compute, the same number of evaluations, this is what you get with MAP-Elites. It’s a complete sea change in what is possible in terms of an algorithm and going and doing a good job of exploring the search space and teaching you what’s possible, like that there’s a really good local optima over here you might never have discovered with another algorithm.

[10:12] Jeff Clune: Also, you can do science on these algorithms to kind of study the properties of things like goal switching. And what you can find is if you take the final champion of a specific spot that’s really high-performing, and you say, “Where were its parents?” You trace back the lineage of its his- of its ancestry, you can see these long, circuitous paths through the search space, much like the path that got us to computers and to, um, microwaves that go through kind of weird, unanticipated places along the pa- along the way. So ahead of time, you would never have known that this was a curriculum that would get you to that l- that final great location.

[10:47] So we put these ideas in MAP-Elites together into this Nature paper in two thousand and fifteen. You can see the grid on the floor under the robot. Ahead of time in simulation, we search for as many high-quality, diverse ways for our robot to walk.

[11:00] Jeff Clune: And then on a real robot that has become damaged, we use a very efficient search algorithm called Bayesian optimization to search through all of these really high-performing but different gaits to find one that still works despite the damage. And in one to two minutes, we have a robot that can continue on with its mission.

[11:16] We also applied these ideas in another Nature paper, this time in two thousand and twenty-one, which was tackling hard exploration problems in reinforcement learning. So, uh, we basically, you know, said w- to the algorithm, “Go and try to get to as many different states in this video game or this challenge as possible in the highest quality way, the most efficient or the most high, highest scoring way.”

[11:39] And people had been trying to solve pr- this challenge, in particular these hard exploration challenges in Atari, specifically Montezuma’s Revenge, for a very long time, many, many years.

[11:48] Jeff Clune: You can see kind of a graveyard of failed attempts, many of which specifically tried to solve this game. And then you can see what Go Explore did, uh, which is basically just blow the roof off of this domain and do a really, really, really good job of exploring. It beats the human world record. We went on and showed that it can solve every single game in the Atari benchmark, which the community had been pushing on for, I don’t know, eight to ten years, or I feel like, at the time. And it solves hard exploration robotics problems better than intrinsically motivated PPO, which is the state of… still the state-of-the-art reinforcement learning algorithm. So this algorithm is very, very different from just, like, taking RL and running it or taking RL and just g- trying to use old-school intrinsic motivation.

[12:30] So I hope that I’ve showed you or, or made you at least a little interested in the idea that QD algorithms are powerful.

Open-ended algorithms and POET [12:36]

[12:36] Jeff Clune: Um, but they’re still stuck in one environment, which is not the grand ambitions that we have for the field.

[12:42] What I really wanna do, what I would say has been my quest on my academic career ever since my first paper here at ALIFE, working with Charles and Chris and Jordan and people like that, is I want to produce what we call open-ended algorithms. These are algorithms that truly innovate forever, and I’m preaching to the choir here in the Artificial Life Conference. But like evolution and like human culture, they never will end. They will keep innovating forever. And I really mean that. I mean, if you ran it for a billion years of open AI-level compute, you would still wanna come back and check on the billionth and first year to see what new things it came up with because it’s constantly surprising us, just as evolution does routinely, as we saw with the COVID pandemic.

[13:25] Jeff Clune: So can we do this?

[13:27] And, uh, one other thing I think is a key ingredient of, of evolution and culture of open-ended algorithms is that every time they solve a problem, that just creates new learning opportunities, new challenges, new problems to solve. For example, leaves high up in trees is a solution to the problem of competing for sunlight with leaves that are low down in the canopy. But now o- once you have leaves that are high up in a tree, that’s an opportunity for giraffes and caterpillars, so that’s a new learning challenge for evolution. But then giraffes and caterpillars are an opportunity for things to eat them, live inside them, cooperate with them, et cetera, et cetera. And so every time we unlock something new, we just get more opportunities le- and we don’t get convergence. For example, in technology, once you invent CRISPR, which was its own challenge, now look at all the cool things you can do with CRISPR, right?

[14:15] Jeff Clune: And so we want algorithms that operate that way.

[14:17] So at Uber AI Labs, we went and we tried to encapsulate these principles in an algorithm. This is one of the first papers to encode… actually tried to search for environments as well as also search for solutions to those environments. So we created the paired open-ended trailblazer, or POET. The goal here is to endlessly generate increasingly complex and diverse learning environments and their solutions all in one run.

[14:40] And so the way that we’re gonna do this is that we’re gonna encode this, the environment, the learning challenge, on a parameter vector also, and then we’re gonna run a loop for environments. We’re going to have an, grow an archive of interestingly different environments, and we’re going to periodically pull one environment out, change it a little bit, and then add it to the set if it’s different, it’s new versus the environments we already had, and it provides learning progress for the agents.

[15:05] Jeff Clune: They’re learning something by working on it. It’s not too hard, and it’s not too easy for them. And then at the same time, we’re going to be optimizing the agents and critically allowing them to goal switch. So they might be the best player in this environment, and there might be an innovation that all of a sudden allows them to invade and replace whatever species is occupying this niche, niche. And so we have invasive species, goal switching, uh, re- adaptive radiations of ideas, et cetera, within this algorithm.

[15:34] Well, what was cool is you can also do science within these algorithms and show that your ideas are working. So all of these blue X’s, POET, are environments that POET invented and solved. If you go back and you take that exact same environment and the same learning algorithm and you try to solve that environment, but you don’t do it through this weird open-ended curriculum, you just directly try to solve it, in every single case you fail.

[15:56] Jeff Clune: So you do need this kind of curriculum to get you there, and we even tried to, like, hand-code curriculum and, like, try to solve it other ways, and we couldn’t do it. So you needed the open-endedness properties. So this is a, a slide I’m very proud to show this room. This is a whole bunch of different environments that are being generated and solved all within one run. We’re no longer using the cheat of different runs to produce this huge diversity of environments. What’s also cool is you can actually sh- look at phylogenetic trees of these environments or these agents, and they have deep phylogenetic branches. It’s not the kind of shallow diversity you often get in ALIFE and evolutionary algorithm systems.

[16:35] So POET is cool because it automatically generates tasks, but it’s still stuck in basically one simple little type of environment that we…

AI-generating algorithms and the three pillars [16:43]

[16:43] Jeff Clune: a little search base that we created for it.

[16:45] So we wanna break out of that. I will return to that subject. But now I wanna zoom way out to the grandest ambitions of the field of AI, dating back to Turing and arguably before. If you think we should make AGI or ASI, which is a, a separate talk that I could give on science and ethics, uh, the question still remains technically how will we get all the way there?

[17:07] And I think that there’s a clear trend in machine learning that I put in a twenty nineteen position paper called AI-Generating Algorithms, and that is that hand-designed components and processes and components get replaced by entirely learned processes and components as you have more compute and you have more data. We’ve seen this time and time again in the field of AI. So if you apply that lesson to the grandest ambition of the field, which is making AGI itself, then you can conclude that the fastest path to AGI is not what the field was doing at the time I wrote this paper, and still is largely, which is manually identifying a problem with AI and then manually trying to fix it and show that it works.

[17:47] Jeff Clune: But instead, we should really embrace the idea of AI that builds AI, recursively self-improving AI, and, um, that will be the fastest path. So that is going to require a lot of compute, a lot of search, to search for better AI, but eventually you will produce a very, very sample-efficient, powerful learner, and we have an existence proof of that in all of you. You’re, for a couple more years, the most impressive intelligence we know of in the universe, maybe months, depending on your timelines. Um, and evolution with a planet-sized computer and three point five billion years produced you, so this can work, and can we do it in an algorithm?

[18:25] If we’re going to do this, I think we simultaneously need to push on three pillars. We need to meta-learn the architectures of the neural net, we need to meta-learn the learning algorithms themselves, and we need to automatically generate the learning environment so that it can learn forever.

OMNI, OMNI-EPIC and Darwin-complete search spaces [18:39]

[18:39] Jeff Clune: And I’ve spent most of the talk so far giving you what I think are some of the key tools that we can use on pillar three to automatically generate environments, and I’m going to continue there for a little while longer before getting back to some of the other pillars.

[18:53] So if we want to really create an open-ended algorithm, one that can innovate for a billion years and keep going, we need a truly vast environment search space, right?

[19:04] So actually, the, the challenge is that when you get into such a vast search space, now you have a needle in a haystack problem. Almost every single environment in your search space is gibberish, or it’s too hard or it’s too, too easy. So how are you going to find the needles in there that are really, really good, the tasks that are learnable and interestingly new?

[19:25] Jeff Clune: Well, we have some decent methods for learning progress. People have been looking… Uh, working on this for a long time, including Jordan, uh, uh. Worked on it for quite a long time. I’m gonna use a method out of my team at OpenAI from twenty twenty-one, which is just change in your ability to perform a task, kind of the derivative of learning, uh, or performance, sorry.

[19:43] The, the real problem, the, the, the grand challenge that I think has stopped the field for a long time is that we don’t know how to define what counts as interestingly new. That word interesting is doing so much work in that sentence.

[19:56] Um, but I think a solution showed up on the scenes around the time of GPT, which is foundation models actually already know what is interesting. They’ve read every single article on the internet with lots of humans talking about what counts as interesting versus boring, uh, and they have a lot of common sense.

[20:16] Jeff Clune: You know? They’re maybe not human level in either of these departments, but they’re good enough that we can basically ask a foundation model, “Hey, I have mastered these twenty tasks. What is the next interesting thing for me to, to learn? Or would this new thing count as interesting?”

[20:30] So we can just ask them. So that leads us to OMNI, which stands for Open-Endedness via Models of Human Notions of Interestingness, with these lovely collaborators.

[20:39] And the general idea here is that we’re gonna try to generate tasks. W-we’re gonna use reinforcement learning, although you could’ve used an evolutionary algorithm, but the tasks that we generate have to have high learning progress, and they have to be interestingly different from past tasks that we’ve already studied, mastered, or tried to learn. But… And we, we get the answer to that question by asking foundation models.

[21:00] And then, of course, we’re gonna simultaneously train agents to do well on those tasks.

[21:04] Jeff Clune: So we wanted to be in an infinitely large t-task space, and so we just said, “We will let the system create any task that you can specify in words.” For example, we’re gonna put the agent in a simulated kitchen, and any task is fair game. So the task might be like, put the kettle on the toaster, wait three seconds, and throw it out the window, for example. If… My kids would love that task. Uh, and so the agent could propose that. Now, one problem is once you can ask for any task, you have to automatically tell the agent if it did any task. So that requires what I call a universal reward function, which can tell you to what extent you’ve done well on any specifiable task. Ultimately, I think a video model would do that. Just watch the video and say how well I did, but that didn’t work at the time of this work.

[21:50] Jeff Clune: And so we just had GPT write code that queried the state of the simulator to answer questions like, “Did the kettle sit on the toaster for three seconds, and then did it get thrown through the window?”

[22:01] So now the system is generating the task, it’s generating the reward function, and if you uniformly ta- sample tasks in this space, you do terribly. Needle in a haystack problem. Even if you use learning progress, you still do terribly because there’s just too many tasks where you can get infinite learning that are super boring, like memorizing the next ten digits of some random number string for the rest of your life, for example. However, if you use OMNI, om- on its own, it starts to invent its own interesting curriculum and start to learn kind of the interesting things there are to do in it, in this task. Now, we measure that because ahead of time, we define certain tasks we would expect a robot to learn to do in this domain, and then we measure the performance of the robot on that heldout set as OMNI is generating a curriculum of open-ended learning in this domain for it.

[22:49] Jeff Clune: What’s also cool is it makes kind of an intuitive curriculum. First it finds an apple, then it learns to pick up and fi- find and pick up a knife, and then it learns to cut the apple with the knife, and on and on and on.

[22:59] Okay, so OMNI, I think is nice because now, uh, we can kind of be in an infinitely large environment, but we’re still kind of trapped in that kitchen. We gave it that one simulator. What we really need for open-endedness is that truly vast search space that I mentioned.

[23:14] And so that brings me to this theoretical concept that I introduced in the AI generating algorithms paper called a Darwin complete search space. And I was trying to think, like, what would be like the, the most open-ended search space imaginable? And I was just like, well, actually, as long as you can run any computable task, anything that you can run on a computer. It could be logic tasks, coding tasks, poetry tasks.

[23:35] Jeff Clune: You could have video games that you import. You could write your own simulator and have to operate within it. You could challenge it to write a simulator. Anything. That’s what I call a Darwin complete search space.

[23:46] Uh, and so the next paper that I want to show you about basically applies omni, OMNI, but uses the coding abilities of modern language models to say, “Hey, the language models can just code up whatever environment the agent should be, uh, performing in.” And since code is Turing complete, and therefore can run any com- any computable environment, that allow- that means that this is also Darwin complete.

[24:08] So now the system is just going to be proposing tasks and then learning those tasks, but writing the code for that task that allows it to do any possible task. Now, for this specific domain, we didn’t have infinite compute in my UBC academic lab, so we restrained it to writing environments in PyBullet Simulator.

[24:27] Jeff Clune: But there’s nothing in principle that would’ve prevented it from doing other kinds of tasks. But look at all the cool stuff it made. So we seeded it with, like, two or three tasks, and all of these are generated by the system. So first it makes these kind of, like, moving planks. Then it goes, like, and tries to kick… You can’t really see because of that. You kick a ball through goalposts. Then it, like, s- spaces these planks out. Eventually, they’re moving horizontal and up and down, and maybe they’re twisting. Uh, they ki- it creates obstacle courses, crashing dominoes, knocking over a stack of blocks. Ultimately, in this domain here, the reinforcement learning was the slow part, not the environment generation.

[25:03] So we couldn’t see, like, this huge explosion of environments, but we wanted to see that. So we said, “Let’s just pretend the agent solves every task the system generates to kind of see what the environment generator will do,” and that’s what you see here, this kind of, like, graph of never-ending environments that are being generated.

[25:19] Jeff Clune: And there’s really cool clusters. It has, like, a whole bunch of, like, you know, kick a ball through a goal, kick a ball through a moving goal, kick a ball off of a wall through a moving goal, kick a ball off a moving wall through a moving goal, and it kind of riffs on that theme. But somewhere else in this search space, it’s, like, creating, like, uh, obstacle courses and buildings to explore. And then over here it creates, like, cluttered, uh, scenes, like a construction scene, and you have to clean up the construction scene, or, like, a bunch of, like, dirty dishes on tables in a restaurant, and you have to clean up the restaurant. So just like the phylogenetic tree in nature, which is simultaneously pushing on, like, the big cats and the aquatic creatures and the reptiles, this thing is going off and deeply exploring different flavors of environments and learning challenges for this agent to learn.

[26:03] All right, so Omni finally allows us to focus on interesting tasks, and Omni Epic allows us to do that in any possible environment, including potentially science challenges.

Genie world models [26:12]

[26:12] Jeff Clune: So this is one of the first algorithms that I would be curious to start scaling up dramatically. You know, I… It probably wouldn’t work for a billion years, but I don’t know what would happen if we dramatically scaled it up. I’d love to see that, see where things break and keep going. And that’s an exciting turning point for me in my career, and I think for us in a, in, in our field, that we’re starting to create algorithms that we might want to run for a really, really long time.

[26:35] Okay, um, I mentioned in, uh, that I… Well, I didn’t actually mention this, but when I first came up with the idea of Darwin Completeness at Uber, I thought… I c- I was racking my brain for what might be a Darwin complete, uh, search space, and at the time, I hadn’t thought about code because AI couldn’t generate code. But I did have a crazy idea that I thought was actually a bit of a career risk to put in a paper because I thought people would think that I’m, like, super crazy, and that was what if you just had a neural net that learned to generate the entire world?

[27:03] Jeff Clune: Like, it, it gives you a, like, a, the pixels of the world. You take an action, and then it just gives you the next pixels of the world. Or maybe it’s also giving you sound and smell or whatever it is. But, like, what if one neural net could just be the entire video game, basically? And then it could be any video game. And that was super wild and crazy. That’s twenty nineteen.

[27:20] And literally five years later, I thought that would, might take decades to make, or I might get laughed out of academia for even proposing it, and five years later, I was fortunate enough to be on a team at Google DeepMind that made the project called Genie, which is now a flagship project for Google, which does exactly that. So Genie is a world model trained on internet video, and you can basically ask it for any kind of environment that you want, and it will make that environment for you, and then you can play it. You can go into that world and explore it, take actions, and it knows how to intelligently respond to the actions that you take, uh, which is pretty wild.

[27:56] Jeff Clune: So the original version of Genie, Genie 1, just could only, like, you give it the picture of this. It knows somehow that you’re the orange agent, and if you hit jump, then it knows that you’re, like, on this little platformer game. And we could get it to work for about one second, and then everything would go fuzzy and weird. But we knew that, that, that we were onto something. We really expected that we would be able to scale this.

[28:16] And so, um, all of the things of foundation models apply in Genie. You know, you can basically scale it up, more data helps, et cetera.

[28:23] And we even showed that, hey, every time we give this thing more compute, it works better. So I think it was nine months later, we had Genie 2 out. We just took Google Compute and scaled up the idea of Genie 1, and now you’re in three-dimensional worlds, and the videos are, like, thirty seconds, and you can create all these crazy different worlds, like you see here, like be a robot, be a sailboat.

[28:43] Jeff Clune: You get, like, beautiful lighting. And then I think about a couple months or maybe a year after that… Oh, sorry. Here’s some extra things in Genie 2. You can take actions that are intelligent, like shoot a barrel or my favorite, slice a balloon with a samurai sword, which sounds super fun. Anyway, a couple months after that, I forget the timing exactly, but Genie 3 comes out. So it’s basically the same idea more or less, more compute, and now everything is, like, almost indistinguishable from, like, a triple A video game, except you can ask for whatever you want, and you can play the system, uh, which is just super wild. So here’s, like, a s- uh, a, you know, a jet ski flying around with beautiful lighting and interacting with objects. My favorite, this, this firefly exploring, like, a Tolkien-esque treehouse world, et cetera.

Pillar two: learning from video with VPT and SIMA [29:29]

[29:29] Jeff Clune: So I just want you to look at these videos and imagine what a learning agent could do learning in any conceivable environment in this Darwin complete search space forever. Like, at some point when we’re just getting into the world where the algorithms aren’t the problem anymore, the compute is the, what we, what we need more of because the, the agents are learning, and the systems are getting rich.

[29:50] Okay. So imagine, you know, that’s Genie 3. Imagine Genie 4, 5 or 10. In fact, the guy who did Genie just founded a company and raised a whole bunch of money to go, uh, pursue this dream.

[30:00] Okay. So I’ve done a lot of pillar three now. How do we generate these environments?

[30:03] I want to quickly talk about pillar two.

[30:06] Um, I’m gonna leave pillar one to the rest, to the neural architecture search community because they’re, that’s thriving. So in pillar two, as I mentioned before, currently the bottleneck in open-ended learning is not environment generation anymore.

[30:19] Jeff Clune: It used to be the main challenge, but now we think we’ve got a pretty good handle on that. Now the real challenge is that agents are still really sample inefficient to learn. So every time we give them a new challenge, we have to wait a really long time for them to learn that challenge. So how are we gonna deal with that?

[30:33] Well, my team at OpenAI, uh, created this project called VPT, the video equivalent of GPT. And the idea here is that we are going to have agents learn how to act from internet videos.

[30:46] So there’s, you know, as I mentioned just now, RL is very slow, uh, to learn a lot of challenges, often because of hard exploration that prevents open-ended algorithms from being off to the races.

[30:56] But there’s tons of videos online for how agents can act- learn to act in the world, and, um, especially of Minecraft, which is where we chose to do this work for that reason.

[31:05] Jeff Clune: What we show in this project is basically if you pre-train an agent to act like the agents in online videos, just like you could pre-train a language model to say words like a human would say after these many words, what’s the next word, what’s the next word. Here it’s look at a bunch of internet videos and then take the same actions the human would take in that situation. As you scale up the data massive orders of magnitude, you eventually start to see the agents being able to zero shot, be able to do really hard skills in these video games, including skills that take, like, two minutes and thousands of human actions. For example, look at the rightmost video here. This is an agent that’s been pre-trained on the internet, and we drop it into this village here and, like, look at how well it explores that environment.

[31:51] Jeff Clune: Rather than just randomly mashing buttons like you typically see in reinforcement learning, this agent knows, oh, I’m in a village. I should go house to house. I should climb stairs. I should open doors. If I see a trunk, I should open it, et cetera. And so once an agent kind of knows those basics of the world, it could much more quickly learn new tasks that you give it in an open-ended algorithm.

[32:11] In fact, we demonstrate that here. So we challenge the agent to learn how to mine diamond tools, and if you don’t do video pre-training, you get no performance, basically. It learns the, the only one skill. If you do reinforcement learning and you reward intermediate steps to diamond tools and you have pre-training, then the system ends up learning skills that take skilled human players twenty-four thousand actions and over twenty minutes to perform on a computer.

[32:38] Jeff Clune: And this is using the same interface that a human does, a mouse and a keyboard. So imagine all the things that you can accomplish on a keyboard. The reason we did this work was to de-risk modern computer-using agents. So if anybody here has used Codex to, like, book them a flight or do anything on an internet browser, this is the— this was the work that, like, proved that that was possible at OpenAI. Okay, I went on to a team at Google DeepMind that scaled up this idea. We did VPT, but basically on many, many video games to produce a generalist agent, uh, that can play arbitrary games. And one, one of the coolest results is if it trains on twelve environments, it is almost as good on the thirteenth environment as an agent that was trained directly in that thirteenth environment. That’s the s— the, the vertical— horizontal line there, which is pretty incredible.

[33:22] And here is combining these ideas.

[33:24] Jeff Clune: So now we have Genie inventing a world and Simma knowing how to immediately go and follow instructions in that world and go try to explore and learn how to do tasks in that world. So we’re putting together the Darwin complete search space and the open-ended learning, uh, the, the VP— like the learning by, you know, pre-training to seed learning idea.

[33:45] Okay, so to conclude VPT and Simma, pre-training enables learning hard exploration tasks efficiently and should greatly catalyze our open-ended algorithms, which I’m really, really excited about.

[33:55] Uh, okay, so now for the remainder of the talk, I want to try to share with you that there’s a really new powerful playbook that you— we can bring to bear on the creation of AI itself.

[34:08] This power— This playbook is very, very powerful.

Automating AI design: ADAS, Darwin Gödel Machine, Hyperagents [34:10]

[34:10] Jeff Clune: It involves combining open-endedness with frontier models and ideas like Omni, and I think it will automate science, including the science of making better AI.

[34:19] So the first work I wanna tell you about is with these great co-authors called The Automatic Design of— sorry, The Automated Design of Agentic Systems.

[34:26] So as you probably are all aware, language models used to be text in, text out. Now we have these complicated workflows behind the scene where you ask a language model a question, it might use web search, it might do like five web searches, it might have an ensemble of five proposed answers, and then a debate amongst itself for what to do before it gives you the answer. There’s some complicated workflow there. We call that an agentic system.

[34:49] And if you looked around, people were designing by hand agentic systems, and they were lifting performance quite a bit.

[34:56] Jeff Clune: But that’s not the way that I think that this is gonna go. That kinda sucks to have to hand design these super baroque Rube Goldberg machines agentic systems.

[35:05] The AI generating phil— algorithms philosophy says that learned compo— stuff gets replaced by… Sorry, hand-designed stuff gets replaced by learned stuff. And so why not just have AI learn to produce better agentic systems by automatically designing them? This is kind of a modern version of genetic programming. And so we’re going to let AI create agentic systems in Python, which is Turing complete, and we’re going to harness the principles of open-endedness to basically grow an archive of interestingly different agentic systems, test them out, see if they’re any good, and, like, rinse and repeat.

[35:39] And what we see is that the original agentic system performs very poorly, but over time, it slowly surpasses all of the state-of-the-art hand-designed systems that existed at the time.

[35:49] Jeff Clune: It gets better and better the more compute that you give it. It also produces pretty interpretable reasonable agentic systems that have good ideas, and if you transfer these things to new domains, they work quite well, uh, because they’re just generically good agentic system designs.

[36:04] For example, like having this debate amongst many different types of experts.

[36:10] We wanted to take that idea even further. So in the automatic design of agentic systems, you have one AI that would make a growing archive of different AIs. But what that means is the original AI never got better. So what we did in the Darwin Godel machine is we let the system modify itself. So we grow an op— a, an archive of interestingly different recursively self-improving AIs, any of which can improve themselves. Then we test it and see if it’s interestingly new or better, and if it is, we add it to the archive, and we rinse and repeat.

[36:39] Jeff Clune: So now the system can not only get better at the downstream task, but it can get better at improving itself.

[36:45] Uh, and what we see is you can look into these systems that have a phylogenetic tree of interestingly different agentic systems. They often have to get worse before they get better. They invent their own kind of curriculum, and, like, make breakthroughs and build on their own stepping stones and breakthroughs, which is all very exciting to see.

[37:01] You can do the science and say if you didn’t let it self-improve or you didn’t let it be open-ended, if you did like greedy hill climbing in this search space, you don’t do nearly as well. That’s these two ablations that are down here.

[37:14] So the, the final paper I wanna tell you about in this arc is hyperagents. So the Darwin-Godel machine was really cool because it could self-improve. But the reason it worked in many ways is ‘cause the task we were asking it to focus on was a coding task.

[37:28] Jeff Clune: And if we asked it to get better at coding, it happened to get better at improving itself ‘cause it was made of code, and so it could… it would also just incidentally be good at improving its own code. However, if your downstream task is financial modeling or poetry or anything like that, that’s not gonna work. So in hyperagents, we literally let the entire… like, the system edit all components of itself, so it can kind of separately work on how to get better on the downstream task and also separately focus on how to improve its ability to improve itself. Um, and that’s what we call DGM hyperagents.

[38:00] And so again, we have a, a phylogenetic tree of interestingly different stuff. Again, you often have to get worse before you get better. If you don’t embrace open-endedness, you don’t do nearly as well, and it, it races past the state-of-the-art in many of these domains.

[38:14] Jeff Clune: On coding tasks, DGM is very similar to DGM hyperagents, but on non-coding tasks you can see a big difference.

[38:20] So if you don’t manually adapt G- DGM to a new domain, it does terribly. If you do manually adapt it, it does fine, but not as well as hyperagents, ‘cause hyperagents can actually really get good at improving itself. And probably the coolest result is that hyperagents can learn good m- meta-agent skills, like how to improve itself on one domain, and then you can transfer that to another domain, you can see here, and freeze it, don’t let it self-improve anymore, and it’s still really good on

[38:49] that next domain. So whatever it’s learning about how to become a good meta-agent, how to make a stream of ever better agents, that skill transfers. The meta skills transfer, which is what this experiment here shows.

The AI Scientist [39:02]

[39:02] Jeff Clune: We also applied this to continual learning, and it allowed us, uh, agents to automatically learn as they encounter a series of tasks how to write down information about the tasks and intelligently retrieve that information. That worked better than all the manual stuff that people were doing for this kind of problem. I don’t have time to tell you about, but that is in this paper called OLMA.

[39:20] Okay. We’re getting near the end. I wanna tell you about, um, this work here, which I’m really excited about. This is, uh, in Nature in the last year. This is called the AI Scientist. It was done with these wonderful people here. And the idea behind this paper is we’re like, “Okay, we really wanna embrace open-ended algorithms.” The most open-ended domain of all time is the one that everybody in this room is excited about, which is science itself. Can we have science begin to automate… Sorry.

[39:47] Jeff Clune: Can we have AI begin to automate scientific discovery? Can it do the entire arc of work that is involved in one scientific paper?

[39:54] And we ended up finding that the answer to that question is yes.

[39:58] So, um, for this paper, we focused on the science of improving AI systems, so AI research, because in AI research and ML research, the computer is the world. We didn’t have to worry about a simulator to reality gap and all the problems Lee’s focused on. We don’t have to worry about automated wet labs and the messiness of the real world. However, this same idea could apply to those other domains.

[40:20] So the AI scientist is a foundation model prompted in, w- uh, you know, like an agentic system that we’ve created that comes up with its own ideas for a scientific, uh, paper, designs and runs the experiments for that paper, analyzes and visualizes the results, and iterates.

[40:37] Jeff Clune: If it doesn’t l- if it, like, realizes it didn’t ex- actually do the right experiments or the, the code is a little buggy, it’ll fix that and re- repeat the process. When it thinks it has, uh, good results, it will write the entire scientific paper from the first word in the abstract to the l- or the title to the last period in references. Then it will peer review that paper, and only if it’s good enough will it come back and say, “Hey, I…” to the humans, “I’ve done some scientific investigation. Here’s a human-readable writeup in the form of a paper.” Which is, you know, here’s an example paper here, which is something that I’m not sure that I really thought would happen within my scientific career.

[41:14] One time I was reading one of the papers the scienc- the AI scientist had generated, and I thought to myself, “That’s a pretty clever idea. Like, if a grad student came to me with that idea, I would say, ‘That’s interesting.

[41:25] Jeff Clune: Go do that.’” And I put the paper away. A couple of months later, a team from Oxford did effectively the exact same idea, and it was celebrated by the machine learning community. So this was, like, kind of a natural experiment that showed that AI can come up with creative ideas for scientific projects that are celebrated by the community if they don’t know that it was done by AI, or in this case, because it was simultaneously done by AI and humans. I also think it’s the first time that I’m aware of the kind of human scientists and AI scientists, like, made the same in- independent invention ‘cause they were both standing on the shoulders of the same giants. The idea was in the air. Now, to be fair, that team did a… the Oxford team did a better job of execution, but in terms of creativity, that was an important moment.

[42:08] So we took some of the papers generated by the AI scientists.

[42:11] Jeff Clune: I think it… we had to generate 30, and we picked the top three. We submitted them to an ICLR workshop with the permission of the organizers, and one of the three papers scored in the 45th percentile, so 40— better than 45% of human-authored submissions, and was accepted via peer review. So, um, that’s not, it’s not yet at the level of top-tier human science by any stretch, but, you know, it’s already doing work that’s kind of mid-level work of human science.

[42:38] And what we did in the Nature paper is we a- uh, studied and found that the better the model, the better the science. So basically, as the models are getting better in the background- The papers are getting better without even having to touch the system, let alone trying to improve the system. So I think we’re right around… I basically think we’re seeing the, the, you know, the dawn of a new scientific revolution that’s going to be powered by AI.

[43:01] Jeff Clune: You might call it the AI scientific revolution, and it- we’re just starting to see the first rays of light of that dawn.

[43:07] In the future work, I’m really excited to apply it to different domains like biology, chemistry, material science. But what I’m most important, most, like, interested in doing is making it a truly open-ended system, effectively simulating an entire community of scientists inside of a cluster by having the system not just do one paper, but do a paper and then another paper and then another paper, read its own papers, build on its own discoveries, constantly be running off and doing, like, sim- forward simulating decades of scientific work all in a supercomputer, but grounded in the fact that it’s conducting real experiments and getting real data. That’s when it becomes really, really exciting.

[43:42] So as I mentioned, this playbook is extremely powerful. I’ve told you about the work in white. I haven’t told you about the work in green.

Recursive, the industry shift, and conclusions [43:47]

[43:47] Jeff Clune: That’s… That you can check out out of my lab if you are interested in learning more.

[43:52] Uh, I also wanted to mention that I think one of the most amazing things that’s happening in the entire field of AI right now is that AI-generating algorithms, open-endedness, and recursive self-improvement are arguably the hottest topic in all of AI right now, after being a fringe topic for most of my career. Uh, my co-founders and I, for example, created a company called Recursive, and we went out to Silicon Valley and we said, “We want to scale these ideas up. We think the time is now to just really rapidly sc- radically scale these things up and see what happens.” And we were able to raise six hundred and fifty million dollars, which is a testament to the interest the world has in seeing these ideas pursued. And we’re not the only ones. Anthropic, OpenAI, Google DeepMind, and many, many startups are announcing that recursive self-improvement is their top priority, if not their sole focus, uh, which is just kind of, uh, pretty incredible.

[44:44] Jeff Clune: Uh, at Recursive in particular, we’re trying to scale up open-ended algorithms and AI-generating algorithms to improve all parts of the AI stack, everything from kernel optimization to better optimizers, better language models, pre-training, all of that. We’re trying to automate the scientific method, as I’ve been describing here, to basically have, like, a supercomputer of, uh, scientists that you could just r- like, basically simulate a scientific community, first with AI and then other scientific disciplines.

[45:10] And we recently shared our first results. So we made an open-ended search algorithm inside of our company, and we picked three domains that communities of humans have been working on for many months, if not, in some, in one case, uh, many years. And we had our system try to automatically come up with ideas, conduct experiments, get better on these domains. And in all three of these domains, we got state-of-the-art results.

[45:31] Jeff Clune: Here’s a plot from one of them where it’s coming up with its own ideas o- and building on its ideas and surpassing the then state-of-the-art and getting really, really good performance on this particular domain.

[45:42] So if you are still extremely skeptical, if you are still sitting there thinking of all of the flaws in all of the arg- the, the algorithms that I have presented, I would just encourage you to reflect on the fact that this is the worst they will ever be.

[45:56] And I have lived through this transition. Many of you have seen it as well. In computer science, and especially in AI lately, once something barely starts to work, in a few years, it’s jaw-droppingly good. We went from generating images that looked like, like that in twenty fourteen to basically indistinguishable from human quality, uh, or even realistic images, uh, about a decade later.

[46:19] Jeff Clune: And so I think if you apply this kind of thinking to automating science and open-endedness and AI-generating algorithms, you can see what the world might look like in a few short years, which is pretty, pretty wild.

[46:32] The last slide I think I have in the whole deck is on what I wanna call easy versus hard open-endedness, just to plant the seeds of some fun thinking. I think easy open-endedness, which is still a grand challenge and super hard, allows you to use as much human data as you want. So using foundation models like GPT, using internet video like VPT, using models of interestingness from foundation models like Omni, uh, imitation learning for your agents, all this stuff. Um, now, the— that means you can go faster. You can stand on the shoulders of giant human datasets to borrow from Newton. But it also means you’re biasing whatever you get to be very human-like, right?

[47:08] Jeff Clune: So that brings us to hard open-endedness, which is could we create an open-ended system? Could we create another complexity explosion like we saw on Earth with Darwinian evolution with as little human data and input as possible? That allows us, as I talk about in my AI-generating algorithm paper, to explore the space of possible intelligence that might exist in the universe, uh, all the different ways that you could be really smart. And it’s kind of like space travel or inventing space travel. It’s kind of like the ability to visit alien cultures and see what is their music and math and ethics and literature like. And I think this is one of the main quests that the artificial life community has been on for a while, at least a big subset of it, and is one of the reason why I love the work that’s done by all of the people in this room.

[47:51] Okay, three conclusion slides, and then I’ll take questions.

[47:54] Jeff Clune: So I’ve introduced quality diversity algorithms. They explore well. They create and collect stepping stones. They harness goal switching, and they invent counterintuitive cu- curricula that are really effective. I’ve introduced open-ended algorithms where the goal is to truly innovate and learn forever, like natural evolution and human culture, and, uh, uh, specifically in, uh, also introduced AI-generating algorithms, which is basically using open-ended algorithms, but specifically with the goal of trying to make ever better artificial intelligence, AI that improves itself, recursively self-improving AI. I think that’s the fastest path to producing Truly like ASI and really powerful AI.

[48:32] I’ve introduced Omni, where it allows w- the algorithms to finally allow us to pursue what is interesting and do so in Darwin complete search spaces, which are these vast search spaces that have any computable environment.

[48:43] Jeff Clune: We can do that either in code, like OmniEpic, or in, uh, using foundation world models like Genie. And I’ve introduced the idea that you can use pre-training to kind of like skip the boring part and get right to, like, really fast sample ef- somewhat sample efficient learning with things like VPT and Simma. For example, it took evolution about three point five billion years to produce the human brain, and, like, look at what we’ve done in a couple hundred thousand years. Like, that was the really interesting part in many ways. And so if you’re interested in kind of that last phase, well, with pre-training you can kinda skip to that period and then keep going, which allows us to leapfrog right to the interesting part.

[49:21] Uh, I also think this playbook can automate science, including recursively self-improving AI. I introduced the automatic design of agentic systems that are on Gödel machine hyper-agents and the AI scientists.

[49:32] Jeff Clune: And so I wanna leave you with this question, which is that I think that open-endedness and AI-generating algorithms are the next major wave in AI. In fact, they’re kind of the current next wave in AI. People are actively working on them now. And so I would just kind of encourage you in whatever it is that you’re passionate about and whatever you’re working on to ask yourself, like, what could you do with these ideas? I think they’re very powerful, and we’d encourage you to think about them. With that, I wanna thank the Aleph community tremendously, my former advisors and friends who are in the room, and the organizers for inviting me. Thank you very much.

Q&A: herd bias in interestingness [50:10]

[50:10] Speaker A: Thank you so much for a mind-blowing talk. Uh, questions? Um, let’s see. Uh, many questions. So we, we have a little bit more time because we started late, so we will, uh, take around fifteen minutes. I would start here from the…

[50:27] Speaker B: Hi, Jeff. Thanks for that talk. That was really cool. Um, I have a question about this idea of using foundation models to judge interestingness by basically saying, “I’m, I’m gonna look at what the internet finds interesting,” because there’s everything there. But of course, there’s this idea of herd mentality. You’re gonna— internet is gonna find interesting what everybody finds interesting. There’s gonna be stuff on the internet that’s really cool, but nobody’s paying attention to it. Isn’t there a danger that you’re gonna create a self-reinforcing model that really goes into one direction only and might get stuck there? Because in fact, maybe looking at what the interest— internet finds interesting is not really reflecting the diversity of ideas.

[51:09] Jeff Clune: I love this question. So this question does not keep me up at night. Here’s why. I think that the most important thing about interestingness is that it is a function of history. So if, like right now, if I propose to you, “Hey, I have this idea. I wanna put a phone and a camera together and a music player in a little box,” you wouldn’t find that interesting ‘cause we already have cell phones. But at the time of the iPhone, that was an interesting thing ‘cause it didn’t exist yet. And so what we’re really asking the system to do is not objectively say, “Is this idea interesting in a vacuum?” But to look at the archive, uh, within that run and say, “Hey, you’ve generated these thousands of artifacts. Here’s a new proposal. Is this interestingly different than all of that?”

[51:54] Jeff Clune: And then you do that again and again and again as you build the archive. And so, uh, I think that that will allow the system to push way beyond current beliefs of what counts as interesting, ‘cause soon there’ll be new types of mathematics and new types of inventions. The whole system will have no reference point in terms of like, I read an article that said this is interesting or not. It’s gonna have to be a function of its own history and have that general function of new versus what came before. Now, that said, I do agree that because we’re training on human data, we’ll get human biases. It’s possible that, like, we would ask the system a question and say, “Is this interesting?” And some other cool aliens would say, “That’s interesting.” And humans— uh, the, the system trained on human data would miss it and say, “No, that’s not interesting.” But you don’t have to bat a thousand, right? Like, as long as the system, like, can recognize like, like, like twenty percent of the actually interesting things are interesting forever, that algorithm could be open-ended and run for a very, very long time and make a lot of progress.

[52:49] Jeff Clune: So in the limit, you would like it to have, uh, you know, this is like the hard problem of open-endedness, to maybe move beyond anything that’s based on human data. But I think you can go probably really far with such a system. And of course, you could also try to fine-tune it as it goes. Like, as this thing’s making more and more breakthroughs, the really exciting thing is how do you update the interesting model? Like, once it’s in wildly new types of math or like inventions and whatever, it should start to look at its own history and be like, “I didn’t think this was interesting, but it actually ended up being promising,” or, “I thought this was really interesting, and that was a dead end.” And it could start to try to find, like basically do RL on its own model of interestingness. That is a super hard research question, the kind that people in this room could help tackle, that I think is like really, really kind of next level and fun.

[53:32] Speaker B: But you don’t have all of this, right? I mean, you’re buil- you, you’re gonna build up the history, but it’s not like we have a way back machine that give, will give you the internet glimpses of maybe nineteen ninety-four. But I mean, so th- there’s a good point I didn’t think about. So you’re not just taking a snapshot of today to measure interestingness, but you go backwards and say, “There was something today that was not really interesting or people didn’t think was interesting, but maybe, or obviously later it was judged to be interesting, and we need to figure out how did it come about and so on.” Um, so that’s an interesting point. But again, you’re gonna rely on having a really good history-

[54:08] Jeff Clune: Yep

[54:08] Speaker B: … of, of inventions, in a way.

[54:10] Jeff Clune: Well, that’s true. But within your n- your, the new runs we’re running, we have full history, which is also interesting. So the system could look at everything that’s happened and what ended up panning out, what looked like a, was an exciting paper that went nowhere, and start to kinda update its model based on its own data. That’s— We don’t know how to do that yet, but that’s super fun research. Thanks for the good question.

Q&A: open-endedness from nothing [54:30]

[54:30] Speaker D: First over there.

[54:32] Speaker C: So, um, first I just wanna comment. I’ve, I’ve been a strong proponent of this whole direction of research and AI in general, but I have never before experienced an existential crisis, so thank you for that. Um- [laughs]

[54:46] Jeff Clune: I apologize for the latter, but, uh, I’m glad you’re enjoying it.

[54:49] Speaker C: Um, no, no, I’m getting better. I’m feeling better.

[54:51] Jeff Clune: [laughs]

[54:51] Speaker C: Um, so the, the actual question I’m, I’m interested… This all seems to bootstrap off of the wealth of human knowledge and, and it’s like, uh, sort of using that as a springboard, but we ha- the, the sort of questions that I’m interested in, especially related to open-endedness, are before that, is how you, how you start from nothing and get there, and I’m curious if you could reflect on, on that. Obviously, the scientific investigation could help with exploring that, but if you think that those two ideas are related or if what you’re talking about sort of comes in after.

[55:24] Jeff Clune: Yeah, that’s why I wanted to put this slide up because basically at some point, once a foundation model started to work really well, I got really excited, as you can see in, like, the latter third of the talk, and, like, the last five years of my career, maybe more, of using these new gifts fr- uh, from the AI gods to, like, skip past some of the grand challenges in the field, like what counts as interesting or how do you have a universal reward function, how do you have sample efficient learning, and take advantage of that to basically work on the easy problem of open-endedness. But I’m trying to be very clear, I still find the hard problem super fascinating, and that’s kinda what you’re talking about and what a lot of the people in this world are, this community are working on. Like, a lot of the work in Evita obviously isn’t pre-training on human data.

[56:09] Jeff Clune: It’s trying to create a complexity explosion amongst much, much simpler types of organisms. Um, and I worked on that for many, many, many years. I’m fascinated by it. I think that it’s really, really hard, and so I want people to keep going. It’s some of the most fascinating science we have. And yeah, it’s not gonna benefit from the, a lot of the foundation model type stuff. I think it could still benefit from a lot of these insights, like for example, the idea that unlocking any one problem should, you know, create new problems, uh, that we don’t want to have, like, really myopic, uh, single objective functions, ‘cause that’s not gonna lead to great progress. So there are some lessons in here for that hard open-endedness quest, but much of it is not relevant. And I basically think we’re gonna tackle green first, and then we’re gonna have to come back and tackle hard open-endedness because it’s just harder.

[56:59] Jeff Clune: Yeah.

Q&A: persistence and diversity of research agendas [57:00]

[57:00] Speaker D: Yeah.

[57:01] Jeff Clune: Or-

[57:01] Speaker D: Yeah. Thank you, Derek. Amazing talk. Yeah.

[57:03] Jeff Clune: Thank you.

[57:03] Speaker D: So, uh, I would like to bring up the, the issue of the individual differences. You know, probably you remember the time, like, uh, thirty, forty years ago, neural networks, AI was the totally uninteresting topics that if you present neural networks, you are done, right?

[57:17] Jeff Clune: Yep.

[57:18] Speaker D: But there are people who still insisted because to those people, including like, uh, people like, uh, Geoffrey Hinton, they persisted because they believe in that.

[57:26] Jeff Clune: Yep.

[57:26] Speaker D: And that was a huge minority.

[57:28] Jeff Clune: Mm-hmm.

[57:28] Speaker D: But thanks to them do we have this, right? So how do you promote that kind of individual differences, even if you receive the negative res- uh, you know, evaluation one after another? There must be someone who is persisting. So we need to have that kind of thing for the open-endedness, isn’t it? What’s your think, uh, thought about it?

[57:45] Jeff Clune: I love that question. Thank you for that. I also wanna tell a bit of a personal story, which is that I, myself, and probably many people in this room have also lived this. So it’s true that neural nets were like, you know, anathema, and like, like Yoshua, and Yann, and Geoff were kind of like beggars walking through the desert in, like, rags with no water. But to some extent, when I was going through grad school, I hope people don’t mind me saying this out loud, but, like, the same was true for evolutionary algorithms. Like, I was trying to get a job as a, a young academic, a brand-new professor with an evolutionary algorithms background, and it was effectively like having a scarlet letter on my shirt. You know, like I, I didn’t get many top-tier inter- I don’t know, I don’t know if I got any top-tier interviews and, um, it was hard sledding. But we pushed through, and we got a b- like, you know, we were part of, like, getting evolutionary algorithms into NeurIPS.

[58:33] Jeff Clune: I think I was o- like, one of the first modern evolutionary algorithms papers in NeurIPS. Um, and now evolutionary algorithms are one of the hottest topics in all of AI, and almost everybody’s doing kind of evolutionary-based search in the kinda work that I’m showing you, right? Like this recursive self-improving open-ended archive stuff. Anthropic, OpenAI, O- DeepMind, they all have teams on it, and they’re all working on it. So anyway, I just think that’s kind of fun that we have also gone through a similar, uh, story within this community. But to your question, I think the answer is diversity. And so what you want is you want within your search algorithm to be pushing in a sustained way on different branches of the phylogenetic tree. And even if one branch of the tree is racing ahead and performing really well, like support vector machines were when deep neural nets were looking like a dead end, there were still some passionate people that were just making…

[59:25] Jeff Clune: They were pushing in that direction. They were looking at the current stepping stones in that corner of the space and making them better, and better, and better, and sometimes bringing in ideas. But they wanted to keep pushing in that direction. And so that’s one of the most amazing things about evolution is it simultaneously pushes on, you know, ants, and cheetahs, and koala bears, and the fastest cheetah does not preclude the fastest ant. And so in an AI s- in a community of AI scientists, like in the, the, the AI scientists, we do open-endedness for scientists. We want people like Hinton and Yoshua to get passionate about a certain topic and pursue it, even if it doesn’t look like the most promising thing. And we also want people to pursue the most promising thing, so that you want basically like a community where different scientists are pursuing different things and with different styles. Sometimes scientists might be really incremental and just trying to, like, do, like, tuning around the edges and combining ideas.

[1:00:15] Jeff Clune: Other people might be radical blue sky thinkers that only wanna push the limits and do blue sky stuff. You kinda want all of that in your system all at the same time. Uh, which I think is, uh, really cool. And you can specifically reward that, ‘cause with foundation models, you can prompt them. Like, “Hey, all you care about is this topic, or pick one topic and stick with it for a century, and don’t give up no matter how terrible it looks.” You know? And you can take another foundation model and be like, “All right. You’re like a really, like, exploitative person. You always want to be working on the hottest thing. Jump around to whatever’s hot.” You know? And you can even fine-tune the model to, like, only care about one topic. And so you can create this kind of, like, diverse set of, like, weird scientists in your … Simulated scientists that have different agendas and biases and pers- and, and whatever. So we get all the different, diverse kind of minds, you know?

[1:01:00] Jeff Clune: Like, I’m looking around this room and enjoying the fact that I know many of you personally, and you have very different styles of science and different personalities, and we want that in our system. And Jeff is a great example of that.

Q&A: biological constraints on the path to AGI [1:01:11]

[1:01:11] Speaker A: We have so many questions, so we-

[1:01:12] Jeff Clune: Oh, no.

[1:01:12] Speaker A: We need to maybe take only two more. There was one over there that was, uh … Yeah. First there.

[1:01:18] Speaker E: Uh, uh, thank you for the talk. I have a question about, uh, so this, uh, uh, uh, road towards AGI you presented and the three pillars, how important or unimportant you think is this, uh, inspiration from biology of, for instance, you tie all these environments spatiotemporally in a continuum, uh, and then the agent is subjected towards its own homeostasis or metabolic constraints. So this aspect of biology, do you think it is helpful, or we can just skip it to, uh, while, uh, in the, uh, to, to attain AGI?

[1:01:54] Jeff Clune: Yeah. So I, at a high level, I think that basically you can point to almost anything in biology, in neural, in, like, brains or in outside in ecology, and, and say, like, “I think that is a key ingredient, and that should help our system.” I know in ALife a lot of people do that. They, like, say, “That’s probably a key system. Let’s add that to our system and show that better things happen.” Um, and so I think that we should continue to do that, and, like, I don’t wanna rule out any hypotheses. I think what I would just do is abstract that principle, which is not necessarily about spatiotemporal varying or weather varying or temperature varying or whatever. Just, like, s- stuff should change. Like, if you get… If you basically, you… That’s kind of the idea of goal switching. You want agents to constantly have to move and switch, change environments, and you’re much more likely, I think, to get agents that can be generally adaptable and don’t get kinda stuck on local optima.

[1:02:42] Jeff Clune: They get pushed around, and they end up doing really good things. But that’s kind of like a gloss over, like, a vast literature of people who are trying these kinds of ideas. We played with a lot of those ideas in Evita, and they, they help, but they also didn’t, like, magically produce complexity explosion. So we need those ideas, but we need a lot more than that.

Q&A: risks and ethics [1:03:00]

[1:03:00] Speaker A: So there are so many questions online as well and here, so not to be rude, I will take the last question not to offend anybody.

[1:03:05] Jeff Clune: [laughs]

[1:03:06] Speaker A: And then, uh, Jeff will hang out.

[1:03:08] Jeff Clune: I’m happy to take questions after.

[1:03:08] Speaker A: Yeah, you can hang out. And also, may- if you have the chance to have a look maybe later on Discord, we have a server with questions there, maybe. So I know that you are very concerned also of the risks of, uh-

[1:03:19] Jeff Clune: Yeah

[1:03:19] Speaker A: … and the ethical implications of the work that you’re doing. Can you just briefly, like, uh, give, uh, give us your, uh, your view on that?

[1:03:25] Jeff Clune: Yeah. So, um, well, I’m not gonna pull it up now, but I actually just recently, after wanting to for, like, 10 years, I finally wrote a little essay, uh, that summarizes my thoughts on that. So I might not summarize it very perfectly, so I encourage you to read it [clears throat] if you wanna know things in more detail. The essay’s on my website, and it just basically says why I choose to work on, uh, self-improve, recursively self-improving AI despite the risks. Um, in short, I think there’s… I think this technology, just though it has risks, is going to end up being profoundly net beneficial for humanity. I hope that, and I also think that. Every single day there’s unimaginable tragedy on this planet. I’m sure, and I hope none of you experience, I’m sure some of you have. But, you know, every single day there are parents that are crying over their children and watching them die of leukemia or cancer or some other horrible disease.

[1:04:17] Jeff Clune: There are parents watching their… Sorry, children watching their parents descend into the darkness of Alzheimer’s and dementia. There are kids who are going without food, clean water, simple, cheap medicines that are, that, you know… All over the world there’s, there’s tremendous tragedy. And if we can automate science and radically advance scientific progress, we can basically eliminate all of those ills. We can cure all diseases. I think that we can eliminate, uh, resource constraints. We can create… We can solve climate change by creating materials that allow for clean energy. We can auto, take care of our planet by getting the stuff we want without ravaging the p- planet and killing all the species, which is what we’re doing now. So there’s just this huge, tremendous opportunity and upside on the one side of the scale. On the other side, there are real risks, and I am very worried about them.

[1:05:06] Jeff Clune: But I think that if we work really, really hard, we can minimize them in the hope to get the upside and minimize the downside. At the same time, I also think, potentially tragically, humanity can’t solve the tragedy of the commons. We can’t not develop this technology. We… I think it’s inevitable that we will develop this technology. There’s just too much economic value and military value at stake. And so if the best minds in the world, the most scrupulous people, pause and sit on the sidelines, then what’s gonna happen is that the less scrupulous people or the less safe people will race ahead and develop the technology. So rather than have the best minds in the world sit on the sidelines and say, “I don’t think we should develop this,” I think that they should be a part of recognizing we are gonna develop it, and therefore try to develop it in the safest way possible, and in a way that allows that it to maximize the benefits and help human flourishing and minimize the risks and the downsides.

[1:06:00] Jeff Clune: So that’s a kind of a complicated storm of arguments that I’ve been losing sleep over for 15 years, but that’s where I’ve ended up, is that it’s better to be in the race and trying to make it go well rather than sitting on the sidelines despite the risks. And I actually think it’s gonna end up going net beneficial, although I recognize there’s a meaningful probability of it going really poorly. So that’s my answer. Let’s thank Jeff again.

[1:06:23] Thank you so much.