Invitational Summit on the Future of AI Dignity and Sovereignty.This invitation-only summit convenes senior researchers, builders, and institutional partners…
Transcript
0:06 · All right, everyone. Welcome to the super intelligence for humanity. We have just started. It is 10:00 a.m. on the east coast, early on the west coast, and for the folks in Japan, if you’re up, then congratulations. You’re the winners of today. So this is a gathering of friends who are establishing artificial collective intelligence which we’ll refer to shorthand as ACI often and this is really a mature research direction beyond monolithic models large scale AI
0:35 · agendas that we’re already familiar with and the conference the the point of today was really to gather who are the people who we need in the room to talk about how do we create this shared infrastructure this cognitive infrastructure systems that can preserve tacet expert expertise uh maintain provenence and consent and enable this collective intelligence across human and AI and most importantly under governance
1:00 · and so this is hosted by cognacy a public benefit corporation whose charter advocates for a humanity aligned super intelligence then I’m your MC Trisha Wong I’m a sociologist and ethnographer who’s been studying how humans trust technology and I’m also the CEO of advanced AI society industry an industry association for verifiable AI um which is really to figure out how do we actually even keep AI interactions trustworthy. And so this is an invite only conference. It’s a hybrid format.
1:28 · We’re starting out online and then we’re going to start adding all the folks that are in person. And I I think we actually have a few in-person people on the Harvard side. And so here’s how today is going to work. We have a long day. So it’s a marathon, but it’s a really exciting conversation. We have six sessions across a span of 12 hours, multiple time zones.
1:47 · And with each one, we’re going to open up with a session uh with a moderator where I will introduce a moderator and then hand it off to them who will really treat this not as so much a panel, but think of it more as like a salon style since we are amongst friends and we’re really trying to figure out this the answers to establishing this field of ACI. So, we want to hear interruptions. We want to have, you know, this be like a organic conversation as if we were in person.
2:14 · We want to have disagreements and do these kind of real time interactions and not be, you know, no soft panel questions here. So, we’ll be monitoring the YouTube. So, if you have questions, please leave them there. We’ll be pulling them in. And so the the main objective today before we move into the host or the the host of the day is the first objective is to establish ACI artificial collective intelligence as this mature research direction.
2:42 · And the second is to clarify what are the technical and institutional foundations of the shared context which would include you know semantic coordination and provenence and consent and revocability and governed abstract abstraction exchange. And the third is we’re going to examine why tacet expertise and institutional memory are really very critical and though they’re quite resistant to quantification. So how do we ensure that this type of expertise that is collectively owned?
3:14 · How do we capture in a way where we can also develop trust with the community and that they’re not just based on consent but actually active participation.
3:24 · And lastly, we’re here to produce a concise ACI agenda with a clear list of what are the open problems that we’re seeing as the most highest priority.
3:33 · We’re going to look at evaluation milestones and pilot designs and follow up with a joint paper pathway. So now I want to introduce us to our host of today um the three co-founders of Cognacy who have brought us here. First I’m going to have Olaf Wakowski.
4:10 · and I have a shared interest in studying Peru. Uh we are early on in Olaf’s career who was looking at many of these systems and topics the Peru and it’s a place I also opened up a lab in. So I’m really excited to turn it over to Olaf.
4:24 · Thank you Trisha and uh thank you everyone for being here. Um I want to start by saying that uh this summit today uh and tomorrow uh is deliberately a little bit unusual. Uh so it’s not your uh standard AI conference here. Uh it’s not kind of any any kind of um technical um uh AI launch or anything like this. It’s a it’s a really a working experiment.
4:54 · So we are bringing together people who normally sit in different uh rooms, different uh conversations. Uh so AI architecture, artificial life, my field um especially collective intelligence, security, consciousness, even uh linguistics, human flourishing um uh and I guess cultural memory, how to preserve uh the knowledge of humanity and uh institutional u institutional governance.
5:26 · Um and we’re asking whether uh they are actually looking at different parts of the same problem and how um to address this um and how to address artificial collective intelligence together. Um for me um the problem is this um intelligence uh doesn’t appear only inside individual minds or individual um models. Right?
5:52 · So in biology uh intelligence would appear through cells coordinating into bodies, organisms uh coordinating into ecologies and so on and people finally coordinating through culture, language institutions, um rituals, uh the tools they build, memory and trust. uh electricia would just mention um the history of intelligence on earth uh is not only a history of bigger and bigger brains and it’s a history of new forms of coordination and I think today uh
6:32 · that’s uh I’m very excited to have all the perspectives from everyone um and uh that’s why I’m using this phrase of artificial collective intelligence today and and and we’re we’re talking about how to build it, how to understand it, and how to also do it all right. So, we don’t simply um uh talk about multi- aent systems and we don’t mean it as a loose metaphor for uh just collaboration, right?
7:01 · We we mean the possibility of a new layer of AI infrastructure. So systems that uh help humans, agents um and institutions uh all around the world reason together. Um so preserving uh provenence, respect, respecting consent and uh revocability uh encoding tacid expertise in there and um and really coordinating under uh uncertainty. So so the question is not whether AI will become powerful. it already is.
7:37 · Uh the question is uh whether powerful AI um becomes uh sort of opaque or isolated and extractive um just digitizing the books and uh and burning them right so so that’s the opposite of what we uh want to get to.
7:58 · Um so I come to this from uh the a life artificial life and collective intelligence perspective uh originally and that’s where the central lesson is that intelligence is rarely monolithic like what a lot of people seems seem to be investing in uh very heavily at the moment. On the contrary, it’s distributed um adaptive uh and embodied and and often vulnerable and fragile and uh evolving constantly.
8:28 · And that’s the challenge um uh the the challenge is not only I guess to build more capable systems but also to understand how intelligence really scales without losing diversity um agency, memory um and meaning. So today I hope that uh we can uh be ambitious but also careful. So ACI um is not a finished answer.
8:52 · Um it’s a research agenda and we are here to test it together and and uh that’s why we have all those perspectives on the panels today and if we succeed the output of this summit should not just be a recording or uh a set of interesting conversations. It should really form the beginning of a serious road map.
9:12 · So open problems, benchmarks, uh pilot coll collaborations together and eventually um I’m aiming at a joint paper for us today uh that can help define uh the space and yeah thank you all for for being part of this first step and uh yeah I’m passing it back to Trisha who who by the way is one of the people working in AI governance who started as
9:36 · um as an ethnographer not as an engineer and she she spent years inside communities um in China, Latin America, uh enterprise boardrooms, uh studying how people actually make decisions, uh what they trust and what they don’t. Um yeah, fascinating that to you.
9:55 · Well, Olaf, this is why I’ve been such a fan of your work for so long. You know, even when I was um when I left academia, I kept seeing, you know, your work evolve over time. So, it’s really exciting to be able to be in the same room with you. Um and you know I when I left academia I went into enterprise because I said you know this is where the power lies in terms of where big data was um all the you know intensive focus on quantification and so I saw
10:21 · what happened on the enterprise side when companies assumed that everything valuable was measurable and so when I met Ammer it was really exciting because he’s the one who brought me into this world and he’s one of the rare executives I’ve met in this space where he’s not just you know understanding only the tech, but he actually understands the humans and the heart and the systems around it. And he really understood right away when I talked to him about thick data where I’m like, “Hey, there’s data that is resistant to quantification that can’t just fit in a spreadsheet or can’t just fit in a data warehouse.” So, it exciting.
10:52 · It’s really exciting to me when I first met Ammer, I think it was almost a year ago. Um, he started talking to me about some of the stuff he’s researching. So, I would like to introduce Ammer. uh he is really like I said on the applied side one of those rare executives who can who is taking all this frontier technology and saying what does it mean for businesses so he’s been evolving that frontier research and then implementing it in the real world so I want to um you know Ammer please uh please give us a few words following Olaf.
11:24 · Yeah, thank you Trisha and thank you Olaf and thank you everyone for joining this uh crazy uh mission that we are embarking on and and uh essentially it comes from a lot of dialogue with with people like Olaf and Trisha and and Owell and and many more that are joining this conversation today.
11:42 · Uh I was at Nurips last year and walking around the room and it just looked like it felt like the research is stalling uh and and you know the AI the potential for AI is it just so much uh out there and then many of the research I spoke with uh they were just getting tired of the the same old same old and just small incremental improvement over the existing paradigms and uh there were a
12:10 · few questions that were that were consistently being asked you know not not so much into the paper presentations but in the hallway conversation it was around you know why are we building these things what is the purpose what is our role in society um what is the
12:26 · impact of these technologies not just on earth but on our future generation so there was an interesting very interesting exist almost like an existential question in the room right and uh the second part of the question was also around who does today’s AI represent and and and and I think that’s a quite a profound question.
12:48 · I think if you look at the data that has been sourced in to train and we talking AI here loosely and largely talking about uh modern paradigms LLMs and things like that and and then the representation essentially was another big issue that was being discussed and and we all understand that the corpus for today’s training is quite shallow.
13:08 · It’s very low se signal density and it doesn’t really uh uh represent the the uh the the the the diversity of the planet like the life on the planet right and our customs and social and and cultures and languages and practices and actions right so that leads to um the other bigger problem is which is uh in grounding who is grounding it who’s grounding it’s it’s based off of right so that means there’s a lot of perspective uh pushing uh and that is happening right now.
13:38 · Uh and uh the third question that came up was around alignment. So alignment right now is still fairly fairly uh post-training model adjustments and behaviors, right? Things like that. And whereas the alignment just does not represent hyperloized context.
13:55 · Uh an alignment of a of a farmer’s uh uh uh perspective on dryland agriculture from Tanzania versus uh versus parts of Africa versus other northern Africa versus parts of uh even even in the Midwest of the US. Right?
14:13 · So alignment perspective was was being discussed uh uh uh extensively in fact again in the hallways as the missing missing layer and that kind of gets into the governance right and lastly the the question around monolithic systems as they’re evolving right again what I learned from from Olaf is is also that nature shows us that again and also from an evolutionary biology that nature evolves the pro process of evolution is
14:45 · distributed like we learn from a varieties of things right and then and then the evolution is distributed and whereas today’s approach to intelligence you’re not so and then that kind of leads us into the the aspect of collective intelligence so if the intelligence is collective and not just collective in the sense that is is the society as a collective but also at individual level we we should think about at individual level every human expert we are const constantly observing and experiencing life, right?
15:14 · As as we move through it and we tapping into a varieties of domains, whether we are doing that intentionally or or or unintentionally.
15:24 · But the output or inference in a complex situation where intelligence or wisdom is required in decision making. This is there is a collective process happening inside like cognitively as well at individual level and there’s an again a collective process that’s happening at the community level and at the society level. So the question that we’re asking from a research perspective is does today’s AI and in today’s paradigm to advance AI represent human cognition?
15:55 · Does it represent essentially not just the cognition but the colle collective of the society on how the cognition kind of evolved toward it and and what is that next era of AI should look like that represents humanity essentially and
16:10 · it and if it represents humanity then it will serve humanity right the the question that researchers are asking why are we building who are we building this for and what is the impact of this right so I think we have to address those questions head on so in a way we want We want to propose this ACI or artificial collective intelligence uh research thesis as a missing governed basically collective intelligence layer right and then within that we have to understand and then shape out what is that u human
16:41 · and AI uh institutions essentially accountability and governance look like.
16:46 · accountability in this case. What is the social contract of human and AI? Like we have a social contract of human to human, but what is the social contract of human to AI? And then those are the the the the questions that are at least on my mind and then I hope that through this uh dialogue uh we can uh uh further articulate these and then and uh uh generate uh research thesis and I’m looking forward to also then taking and these research thesis and actually building solutions uh that that serve humanity. Uh back to you Trisha.
17:22 · Oh, you’re on mute I think. Let me unmute myself. I’m gonna have to get better about that throughout the whole day. Uh so I’m really happy that you met Olaf, someone who matches your level of ambition. Um Olaf talked about how important it is for us to be ambitious but also to be careful at the same time with the fragility of the human life that we are capturing in these systems.
17:48 · And I think certainly um your your take on this is so rare but also I think there’s a growing contingent right of people who are saying this is not enough like we are stalling and I know that this is why you are also in um in a very a shared like
18:05 · mind with UA Kumar who is also co-founder of cogniz innovation ecosystem builder based at Harvard and he’s also working at the intersection of all of these issues especially around governance and some policy and also the actual technology itself. He founded quantum alliance.
18:23 · It’s a Harvard MIT nonprofit that’s focused on getting you know actual technologies that we’re talking about here from lab to market and he’s also worked with the UN FAO and the German foreign office and he’s just been at the center of all these places that we’ve all been around but he’s actually you know bringing all these places together.
18:42 · So I’m really excited to turn it over to Kumar. I know you had some words you wanted to say to follow up um on this new kind of ambitious research agenda that we’re outlining today. Turning it over to you.
18:54 · Thank you so much Tracia. So I I want to do something slightly different with my few minutes. Uh Olaf just laid out the scientific arc. ML like framed the SEI thesis very well. I’m not going to restate either of those. Instead, I want to tell you what brought me personally to this room. I have spent a good part of my career inside institutions that are supposed to coordinate knowledge at scale. The UN system, foreign ministries, universities.
19:29 · And the thing I keep noticing over and over is a gap that nobody quite names. We have models that can pass the bar exam. We have systems that generate code, summarize research, draft legislation.
19:45 · And yet, if you ask a basic question like who decided this model could train on a on that community’s knowledge, can we verify that? Can we revoke it? The answer is usually silence. Not because people do not care. People do care. We have seen it. But the infrastructure to answer those questions does not exist yet. This is what brings us here. This is what this summit is about for me. Not bigger models, not faster inference.
20:17 · The missing layer underneath all of that, the institutional infrastructure for accountability, for consent, for for provenence, for collective coordination between humans and AI systems. And I want to put two words on the table that I think matter for what we are building.
20:39 · The first is dignity. And I mean something very specific by that. I mean that a person’s knowledge and expertise are not just raw material for the system to ingest. A farmer in West Africa whose family has worked the same soil for generations. A surgeon in Mumbai. A climate modeler in Sao Paulo. The expertise carries authorship.
21:06 · It carries context. It carries decades of embodied judgment. And if we build collective intelligence that treats all of that that as a training data, we have not built intelligence. We have built extraction. This is what I personally feel. The second is sovereignity. Not just data sovereignty in the compliance sense.
21:28 · I mean the question of whether communities, institutions, nations actually get a meaningful say in how their knowledge is represented and governed inside these systems not as an sorry not as an afterthought but as a checkbox as a design requirement from day one. These two ideas, dignity and sovereignty are why we named this summit, why we named it super intelligence for humanity. Not super intelligence as a race to the most powerful system.
22:06 · Super intelligence as a collective capacity with humans in the loop with accountability with the ability to say no. So what are we actually trying to do over the next two days? I will keep it simple. We want to leave here with three things.
22:24 · A a joint paper that articulates the ACI research agenda and honestly including where we disagree. A set of benchmarks, real evaluation milestones so that this does not remain a conversation about values with no way to measure progress.
22:43 · And three like so the third one like um like probably a couple of pilot collaborations with names attached and that people in this room and like uh and in this summit are willing to carry forward with at the uh the session ends. This is what turns a summit into a working group for me and this is the bar I hope we hold ourselves to. One last thought before I hand it back to Tricia.
23:15 · As you listen today, I would ask you to carry two questions with you. What is the strongest open problem you hear? And what would you be willing to work on when this is over? Thank you. This is all Tracia.
23:31 · Ah, how inspirational. I didn’t even want you to stop. I mean, you mentioned the words dignity and sovereignty and that is at the heart of what we’re doing. I mean, I think really what you’re talking about, I mean, Emmer will relate is that part of it is that we cannot be deploying AI as if we’re just deploying databases left over from the ad world, right? Like this is we have we have not fundamentally shifted our ways of gathering data and that’s why it’s extractive. It’s left over.
23:59 · Yeah.
24:01 · So with that uh with that really inspiring intro from all of our hosts from Olaf, Ammer and UAL where you really set the agenda on turning this research area of ACI into a working group so we can have real pilots and a joint paper coming out of it. Let’s take a coffee break and let’s come back in exactly 27 minutes at 11:00 a.m. Eastern Standard Time.
24:25 · And what we’re going to do then is we’re going to start the first panel. uh session 1A called architectures beyond scale and we’re going to have Olaf as the moderator. So come back sharp because that first panel is really going to set our agenda with two very very important researchers who will be in the room. See you in 27 minutes live. Okay, sorry I didn’t realize that we were live. Hello everyone.
24:55 · Welcome back. Usually I look for the little live symbol, but I’m getting used to the new interface. I’m Trisha Wong and I’m your MC for the day. Um, you met me earlier when I introduced our three co-hosts, but this is the Super Intelligence for Humanity conference.
25:11 · And, uh, if you are following online, please drop your questions in the YouTube link. And for those of you who are just joining, uh, I want to set up the frame for what Olaf, Ammer, and introduced to us this morning. So where we are right now is that the dominant bet in AI has really been what we’re all familiar with which is just you know bigger models we have to scale we have
25:34 · to have more compute and more data but that bet has produced real capabilities as we have seen but has also left this entire layer underneath that’s still missing and Ammer kind of hinted at that when he said he was last set in Europe and there was all these conversations in the hallway that weren’t being represented on the actual agenda.
25:49 · So where we’re going today with this conference is that intelligence the premise is that intelligence is distributed and it’s coordinated and if that’s the case then we know it’s not uh the current way of these monolithic forms of representing intelligence is insufficient. It’s just the beginning.
26:08 · So systems that represent uh the people whose knowledge actually runs a world or their worlds and their hyperlocal worlds with context and authorship how do we keep that intact? How do we bring that into our new governance systems, our new forms of AI? And that means we have to develop new types of governance that give communities and institutions institutions meaningful authority over their participation. Um, and they have to have revocability rights, all that kind of stuff. So, we’re calling that entire direction artificial collective intelligence.
26:39 · It’s an emerging field and it’s AI that builds dignity and sovereignty, two words that said. And it’s really about building it into the architecture not just as a patch um afterwards which is you know what we’re seeing right now. And so the next 12 hours are about this exploring this very topic and the first session to explore this topic is going to be run by Olaf one of our hosts and think of this session and all of them to proceed uh that will happen is really as a salon not a panel.
27:08 · And so for this session we begin with the architectural question which is if the next intelligence explosion is not just this single supermind but this plurality and we have to build a plural infrastructure well what actually has to be built you know uh what does AI look like when you just don’t assume that everything is one giant model so for this first panel we have two of the world’s leading thinkers in this space on distributed intelligence and curiositydriven learning so we have again Olaf as our moderator and He has these extraordinary speakers.
27:40 · We have Bla1 Aguera Arcas and he’s the VP and fellow at Google where he founded paradigms of intelligence group and he invented federated learning which many of our familiar with but his recent work in nature is of most importance to us and also his upcoming MIT press uh MIT press book about intelligence and how to understand it through an evolutionary lens. And so his work is really his most recent work is is really forms a core inspiration for why we have this conference.
28:07 · And then we have calling in from France we have Pierre Yei who’s a research director at Inia in Bordeaux where he heads the flower scheme and he pioneered this concept that we’ll be talking about a lot which is curiositydriven learning algorithms where agents learn by generating their own goals and measuring their own learning progress. So we’re moving into a world where it’s machine to machine communication. So his learning progress hypothesis has changed how our field how the field thinks about you know exploration to even begin with.
28:42 · And so Olaf, I’m going to hand it over to you to take this conversation where it needs to go.
28:47 · Amazing. Thank you so much Trisha. uh this is uh the first technical session of the uh of the whole day and uh we uh we we hope that this is going to be the inspiration also and uh uh I think we’re going to dive right in and uh ask ourselves a very difficult question I guess for this session which would be uh what would it mean um to design AI architectures uh AI architectures that um are for uh uh this distributed uh sort of layer of intelligence, right?
29:21 · So, uh the dominant story of AI progress has often been told um as a story of scale, especially recently, right? The larger models, larger data sets, larger compute and eventually perhaps uh one very large system that crosses uh some threshold into general uh intelligence. And uh in spite of that many of us here suspect that that’s not the whole story.
29:48 · And in uh inspir inspired by I guess natural intelligence and um in in the field of artificial life we look at that a lot. Uh the great leaps have not only come from um scaling individual minds but they have come from new architectures of coordination. to sociality uh language writing uh institutions, networks, right?
30:12 · Scientific communities that we’re embedded in markets and and protocols between many many uh agents from from very diverse types and types of intelligences um that allow intelligence to become um collective, right? So, so how do we design uh AI architectures uh to to embrace that and and to implement uh this next layer and and uh I’m very happy to have uh these two speakers with us.
30:43 · So both blaze and piv uh have uh developed very important pieces uh in research, right? So so blaze with the powerful argument uh that the next intelligence explosion may not look like a single silicon brain but like a plural um social and and I know you’ve been advocating for that for a while now.
31:06 · Um and uh and I guess uh Pere um uh with uh the essential piece I would I would I especially like your your your work on curiosity right uh so on intrinsically motivated learning and um autotellic agents so so that means uh how how systems can generate their own goals and I think that’s instrumental to the next layer.
31:31 · Um so let’s dive uh right in and perhaps um uh maybe a first question for for for you blaze uh in your recent work with um James Evans and Benjamin Braden right so so you you argue that the next intelligent explosion is likely to be uh this plural social relational uh uh system rather than monolithic. uh from your perspective, what would you say is the most important architectural shift
32:03 · uh we need to make um to to if if if we take that that claim seriously, right?
32:07 · So so in other words u um what has to change when we stop designing for the most one uh powerful model and and start designing for uh society of minds.
32:21 · Um great great question. So I mean one one um I guess slight twist on the on the question Olaf is that I I actually feel like even what we think of as today’s monolithic models are not actually monolithic when you look inside them. So you know our argument in that piece is actually um uh two-sided.
32:37 · Uh it’s it’s on the one hand an argument for scaling not just of single models or data or of compute but scale but social scaling um uh you know as being as being the basis of say collective human intelligence.
32:54 · Um and this is just an an observation from from anthropology and from the social sciences that um you know what we tend to think of as human intelligence is not really the intelligence of individual human brains but it’s actually um you know the a kind of collective output from human society as a whole. You know when you say um you know in in what sense are humans the the most intelligent animals on earth?
33:18 · Um, you know, if you look at us in our uh in our native state, if you like, you know, an individual person uh you know, raised uh like Miy uh you know, would actually not stand out that much relative to the other uh large primates. But of course, when you when you get a few million of us together, we can do some pretty impressive things. Uh you know, we put a we put somebody on the moon, we we we can transplant organs. Um you know, but those are not those are not the achievements of of individual people.
33:49 · there is no single person who knows how to transplant an organ or get anybody to the moon. Um and in fact if you if you ask most ordinary people um how a how a bicycle works they won’t be able to you know answer there there’s very famous and funny uh publication uh from a number of years ago um in psychology uh
34:06 · you know that is basically like okay draw where the chain goes on on the bicycle and and most people can’t do it um you know uh you know even if they ride the bike every day um right because we have this incredible division of labor cognitively um so it’s a societal level uh phenomenon and uh and you can also see evidence of This for instance when you have very small societies um you know the the way Tasmania got cut off from uh from mainland Australia uh when when sea levels uh changed uh you
34:35 · know and and then like the Tasmanians have lost a lot of technologies that that um that that that the population had had when they were when they were uh joined with Australia or the way uh you know in in some of the in the north in North Sentinel Island for instance which is a very small isolated human population today uh people appear to have lost the ability to make fire. Um you know and which is an extraordinary thing.
34:56 · Um you know so human intelligence is collective uh is the first observation but but the other observation that that goes into that into that paper is that even if you look at a single large reasoning model um there there’s actually strong evidence that what’s going on inside that model looks social too.
35:15 · Uh in other words that there are sort of little voices inside there that are kind of trading off against each other. um you know and and um and that the way reasoning works is in some sense like an adversarial collaboration. You know that there that that that these uh this internal argument is happening when we talk about you know having um an argument with ourselves or you know thinking about something and going over the pros and cons or whatever. You know I I I think that we what we start to see is that that may literally be the case.
35:42 · That really is what is going on. Um so we’re making an argument for social intelligence all the way up and all the way down. Um what does that mean in terms of of architectural changes? Well um in some sense we already have a lot of architectures you know ex in existence that support this kind of sociality. when you look at what soft max does, what attention does, um you
36:06 · know, what lateral inhibition does in the brain, you know, I think a lot of that is about supporting the ability for multiple sub intelligences, multiple cortical columns or whatever it is, um or multiple, you know, sub networks uh in in in a neural net to be able to solve parts of a problem uh call for attention uh you know, when they’ve got a part of a solution, uh etc.
36:24 · But um uh but if we think about about how to how to how to really scale this up and how to do this at at uh at social at social scale um I think neuroeconomics is a big part of the missing uh of the missing problem. Uh what I mean by that is um making energy explicit uh as a as a way in which parts of models interact with each other. Um you know so energy today the proxy for that of course is token budgets.
36:52 · Um but uh you know if you if you introduce economics uh you know into into models themselves I think you know you you form the basis for some of that kind of of self-organization to happen in a way that actually has stakes.
37:07 · Those are amazing elements to to get us started and I agree that uh this neuroeconomics uh the the the way we can trade uh information between uh between those different parts uh of this collective intelligence is really crucial. Um I really like also that so so some of my work recently is on the ID so the dissociative um identity disorder uh and and how we can understand that facing a latent agency within those large models as well. So so I’ I’d love to dive more into that as well.
37:39 · Um also Keriv uh your work uh also addresses directly uh some of that. I wonder about your thoughts on how collective uh agents uh can um exchange but also with that exchanging information between each other. um how uh exchanging knowledge with each other they can also explore uh open-endedly
38:05 · um and uh in a curiositydriven way uh which is some core part of your work right uh and that would suggest that uh that intelligence can develop maybe uh exploring in new patterns if it’s collective uh and then how to get it to be uh for this exploration to be unconstrained uh and I think uh yeah there are many consideration but what does it look like uh to embed this curiosity in in collective agency.
38:36 · Yeah
38:36 · thank you very much for the for the question. Um so maybe as as a first remark uh you we started the question the session about the question of um uh what are the architecture that we need for either individual or collective intelligence and I believe that um asking this question that way uh in a way reveals uh some kind of current bias of artificial intelligence.
39:04 · architecture is a word that that is very static and I think it reflects what is being done in AI currently which is uh large scale um training of very big models so that once the training is finished they’re released and they become static and they don’t learn anymore when interacting with users for example well you can learn a little bit through context learning of course but That’s not a really very flexible and open-ended kind of learning.
39:36 · That is extremely different from the way humans learn. Babies learn uh human babies but also the babies of other species. Um they learn continually new things
39:52 · all their lives but especially at the beginning of the life under very strongly limited resources of time of energy and they are not living in parallel computers when they can actually explore at the same time many things they have one life in one physical universe so they need to really use their time uh the the best they can and uh I’ve been working a a lot for many years to try to understand what are the mechanisms that enable what I think it’s the true form of intelligence.
40:21 · The fruit of of intelligence is not so much to to to know a lot of diverse things but then being static. It’s it’s more about being able to learn very efficiently something new. And there are two ingredients which I believe are absolutely core. One is curiosity and the other one is social learning and and and an associated notion of collective intelligence. And so what’s curiosity?
40:50 · Um, curiosity is is basically a term that designs a a set of processes that push humans to spontaneously explore their environment, their body, the interaction between their body and their environment. And there is one particular form of curiosity that we’ve been studying.
41:05 · It’s what we call autotellic curiosity, which is the ability uh of humans to generate their own goals, their own problems, their own games. um uh always trying to push a little bit beyond what they’ve been exploring so far, increasing progressively complexity but in a developmental manner going just beyond what we know, not too difficult, not too easy, not too difficult.
41:30 · Um and so we’ve been working on various models showing that actually when individuals uh um generate goals uh that can be sensory motors they can be more abstract for example le leveraging language as a cognitive tool um u so that can lead
41:53 · individuals to open-endedly develop continuously new skills and what’s extremely interesting is that this curiositydriven individual develop has very interesting and rich interaction with the social level with collective intelligence. Um the the on the one hand
42:12 · um um one of the most powerful tools uh that humans and children in particular use to invent new new new goals, new games is language. language as an abstraction uh engine uh that is initially learned as a communication tools but as Vigotsski uh a very famous
42:32 · Russian psychologist explained which is internalized uh in such a way that it becomes possible for the brain to simulate social peers and to use language as a combinatorial engine uh to imagine new goals to imagine plan for achieving those new goals to self-evaluate how good one is to achieve those those new goals. So to self-reward towards self-generated goals. Uh so that’s an example of how a culturally evolved system language can influence individual exploration and innovation.
43:04 · But at the same time it is the the innovation of each particular individual which is leading uh to the discovery the various discoveries that that are made by humans and that are going to inspire back and push cultural evolution further in an open-ended manner. The field of cultural evolution for many many years has focused on the question on the evolution of solutions.
43:33 · Uh for example, if you look at at the scientific literature on cultural evolution, you have experiment like chains of interaction where some people do things, some others observe and get and get inspired and transmit.
43:47 · But what they transmit is solutions to problems that are experimentally imposed to them. It’s a little bit the same bias than traditional machine learning or AI where engineers assume that there is a problem or reward that comes outside the organism and the whole problem is about solving that problem. And that’s the same in cultural evolution. But in reality, just like for infants, children, there is never any reward that is given by the environment.
44:15 · There are just physical things like like food like like movements of others like like like physical events. And this is the brand that generates its own intrinsic rewards. And that’s the same at the cultural evolution level. One key question is where do goals come from?
44:31 · How do they evolve? And this is strongly rooted in individual curiosity which then leads to open-ended collective exploration and innovation.
44:42 · Amazing. and the the the emergence of those goals of course is something that we are very active in in in this field uh to to to try to understand the dynamics of and I I wonder if we could address this now in relation with the energy efficiency. So how how does this including the the Vid Gutskin Vid Gutskin frame I guess of well thought being formed from uh language really and and his framing for children as well.
45:12 · How can we couple that um emergent emergent of emergence of goals within agents including humans um in a collective with an economy? So, so can we have something that is more energy efficient for example by being closer to to certain groups of agents realizing certain tasks?
45:32 · Uh is there something something in there uh in in sort of not I I don’t want to say a very good point about uh not not having to engineer um top down the architecture for collective intelligence. I don’t think that makes uh very much sense. Uh but is there a way to do a bit of both? Right. So so top down and bottom up uh to relate this compute efficient um intelligence at the same time as the autotellic picture. A difficult question for you.
46:04 · Can I can I pick your brain blaze on this?
46:10 · Well, I mean I I think it’s actually very similar to what I was uh saying earlier. And by the way, uh you know, um I agree completely with with everything we just heard about uh um from Piv about uh about curiosity and and and its connection with with uh with with social learning. I I that that’s that all feels very close to my heart as well. Um but uh but yeah, I mean the the the way to um the way to engineer those things in without feature engineer, the point is not to feature engineer, right? The point isn’t to introduce uh um exttrinsic uh rewards.
46:42 · Uh uh you know we we would all like to get rid not not we would all I would like to get rid of of um of externally imposed uh rewards. I mean in some sense the reason that we finally got AGI uh you know as opposed to um the narrow intelligences that we had in the 2010s uh you know handwriting recognition, face recognition whatever is that we stopped imposing an external reward.
47:08 · I mean I I know that that view is not is not universally held but um you know we should keep in mind that when we were doing purely supervised learning. Um the best that those models could do was to reproduce the answers uh
47:23 · you know if we were saying like you know this pattern of pixels is a zero this is a one this is a two you know up through nine um then you’re going to get an you know an emnest digit recognizer you’re never going to be able to have a conversation with an emnest digit recognizer. the very best it can do is to score 100% you know on the on the test data right so so um you know it was
47:45 · when we began doing doing um unsupervised uh training that uh that we that we got intelligences that were fully general uh and and that you could have a conversation with you could you know change the rules of the g you know you say let’s play such and such game let’s now change the rules of the game um you could ask it you know what do you want to talk about now um you know we we
48:06 · um we talk about these about these models having um you know exttrinsic rewards but it’s not it’s not exactly true I mean there have been actually a bunch of a bunch of um experiments uh recently on um you know actually figuring out what what models want you know and you can do that in a kind of force choice setting by by saying um you
48:28 · know why don’t you uh decide which of these two tasks you would like to do you know here here are two and then you do a revealed uh you know uh revealed preferences experiment and uh you know and different models actually like to do different sorts of things. It’s it’s it’s it’s quite interesting to see what the revealed preferences of models are.
48:44 · So uh you know those were not those were not trained into them. Those are those are uh emergent. Uh but um but anyway the the uh the way the way that we can bring efficiency into the into the picture um without uh without engineering it in is to just acknowledge that you know everything that an AI model does costs something uh it costs
49:08 · money uh you know and um uh you know and and money is a proxy for energy uh you know that’s that’s um that’s not something that is engineered it’s the reality Now, you know, as we start to get models that that can self-improve, which is to say, you know, that can um uh change their own architectures, uh you know, develop u new models of their own that they can then run, right? Um you know, that’s actually a really that’s a really nice capability if you think about it, right?
49:37 · Uh you know, you can have a model that um that can uh generate, you know, a new environment, you know, run a cloud instance, uh you know, code something up uh you know, and run it on that cloud instance. Well, you know, the way to make that real, the way the way to make that uh to to you know, both have uh you know, sort of open-endedness and curiosity be a part of that, but also um you know, impose uh pressure for this to um uh not exceed
50:05 · resource boundaries that uh you know that that you’ve got uh and uh and to do more with less is to just make the model aware of and limited by uh you know the the resources that you’ve got. um you know you can’t you can’t generate an infinite amount of compute power yourself. uh you know, you have you have finite resources and and having the models feel those finite resources as well as having the creativity and the ability to uh to make new things of of their own.
50:31 · Um you know, unlocks a lot of the a lot of the forces that have led to human social uh diversification, division of labor, uh growth of social intelligence, right? I think you know the reason that we you know have a blacksmith in the village rather than everybody making their own horseshoe is because it’s more efficient uh when when one person gets really good at making at making horseshoes and then you know can uh can give them to everybody and in return is is uh you know uh is able to buy their food from uh you know from
51:01 · others because they’re getting paid right and and I’m not saying that economies are perfect or that capitalism is the way to go but you know no matter what the economic system is that you’re working with right there’s always this issue of uh you know specialization uh bringing uh economies of of scale in in environments that have finite resources.
51:20 · So um you know I I feel like I’m repeating myself a little bit you know with my neuroeconomics point but but the you know my what I’m really trying to say is that that social scaling and open-endedness and and and creativity and social rewards actually all go together. They’re all part of the same puzzle.
51:37 · Excellent. the the it it might be that uh some of the the discovery of goals uh collectively by by agents um and and them having different preferences that you were mentioning uh may make it uh at the end of the day more efficient. I wonder uh what what do you think uh P is um is curiosity uh enabling some energy saving in terms of uh problem solving um in the economy?
52:10 · Yeah, that’s that’s actually probably the reason why why it evolved. Uh what we’ve been studying a lot uh for many years um uh are the the outcomes of uh um uh learning driven by curiosity. And initially um when we started working on curiosity
52:31 · we did it not so much from that perspective rather from the perspective of trying to model the way uh the developmental trajectory of children is structured uh and and and um and uh for example how to how to explain those different stages through through which they go which is a kind of curriculum if you want if you take the perspective of education at perspective of of machine learning. But that was not initially what we developed.
52:58 · But then we quickly um uh understood uh that one of the of the key problems that infant face when they explore their body uh that it’s what I I said earlier is that that’s a very high dimensional body in a very high dimensional environment with so many things that could be learned with also so many other things that are impossible to learn or for which it’s possible in
53:24 · principle to learn them but within the lifetime it’s impossible to learn them in practice And so the real question is how to to explore to to to to collect data in that very complicated space within such a small amount of time and energy. And this is actually what led us to develop this theory of learning progress which we’ve been we’ve been calling the learning progress hypothesis which is the idea that the goals the learning situations that children select in priority to explore are those with
53:54 · the highest learning potential. those that are neither too easy nor too difficult uh and for which the brain believes there is a possibility to improve. And then we’ve we’ve been discovering that not only when you run a simulation on a robot for example with such a mechanism you reproduce the structural properties that are observed in child development both the regularities and the diversity of of trajectories.
54:21 · But you also have an automatic curriculum uh of exploration and learning that has quasi optimal properties in terms of learning with limited resources of time and energy.
54:35 · And what what we’ve discovered is that there are many difficult problems. For example, imagine you have an engineer who has a problem that he wants the robot to solve that that is with very rare or sparse rewards. Then if you use it uh very standard uh optimization techniques uh like for example deep reinforcement learning techniques uh that are going following the gradient when there is a gradient that you can observe and when there is no gradient at all they just do random exploration.
55:07 · Then in in those cases mhm when the food for example for an organism or the problem for a robot is very difficult to find most of the time at the start all the time actually there is no gradient signal at all. So what typical approaches do is that they do random exploration and when you do random exploration in such environments.
55:27 · You need basically to wait millions of years before by chance you stumble on a policy that gets you a little bit of reward and then you can do gradient descent and and this works. And so instead of doing random exploration toward towards uh hoping to to find information about one problem, one specific problem, it’s much more powerful to selfgenerate other kinds of problems uh and explore by curiosity in the environment in a structured manner given driven by learning progress.
55:56 · And because there is structural coupling in the environment, this will statistically lead organisms to make discoveries that end up being useful to solve the difficult problems. That’s why for example animals who live in environments where finding food and reproducing is extremely difficult and environment is changing all the time. So you cannot learn a fixed strategy uh through evolution.
56:21 · The best strategy is actually to forget food, forget mating for a while. Be curious about envir environment and this is the way to be efficient. And I believe that uh such um organized exploration is also something that is very relevant at the collective level.
56:44 · We’ve been working also at at at understanding what is for example efficient collective exploration for problems where you have sparse rewards many local minima. And what we found is that a little bit like curiosity quite often what’s very important is that those individuals in the collectives not only they they they
57:09 · have to forget a little bit the the common objective but they also need to explore their own diverse ways and not to share too much information with others because if all information is shared all the times to everyone then everyone falls very easily very quickly in local minima.
57:26 · And so that’s why for example if you try to optimize the information uh the topology that enables information sharing what’s in general much more efficient is a dynamic topology something that alternates between very little connected collectives and sometimes you you reconnect and then you disconnect and it is actually very efficient from an energetic point of view.
57:53 · Right.
57:53 · I I I wonder uh so so so I wonder what you think about this blaze but also what do you both think about and we’ve all gone through u in I guess all those multi- aent systems simulations optimizations uh adversarial patterns uh
58:12 · in which um some agents become freeloading on the economy that has been emergent or that you have designed um and uh of course uh niche construction uh enables uh more discovery and so so Terry Deacon and others have been proponent of that and we we’ve seen tremendous results in the field for this but also how to avoid um basically viruses some of some of those um agents that become aware of uh free energy within
58:43 · the language of the system at this level of niche construction uh being available for them uh for free and uh them taking advantage of that uh I think this is also key. Can we get away with something that is not cryptographic but that is trustbased? So so I know this is a challenging question. I there are some tools in the field. I wonder what you think about how to address that in a collective uh kind of way to implement intelligence. Uh blaze first perhaps.
59:13 · Yeah
59:13 · it’s a great question Olaf. I so first of all I mean I I think that um misalignment is actually crucially important to collective intelligence uh for a lot of the reasons that that behive uh just uh talked about. You know he he framed this in terms of of of u of information shielding or uh I guess what Friston might call marov blankets.
59:32 · You know that that um you you don’t want everybody to know about everything uh that everybody else knows. uh you need to actually you know um conceal information as it as it were in order to be able to try something um out of distribution uh and and for everybody to not sort of collapse onto the same solutions to things. Um you know the there there are um similar findings uh by the way uh you know with with respect to uh adversarial relationships uh in conversations.
1:00:04 · So uh you know my my co-author James Evans um has uh you know done a bunch with with with his students and and some other collaborators has done some very interesting work on looking at uh at the kinds of of um relationships that that uh people have with LLMs and also that these voices inside LLMs have with each other. And it turns out that if they’re misaligned the results are better than if they’re aligned uh for exactly the same reasons.
1:00:32 · uh disagreement is actually much more constructive than agreement. Uh if you uh you know if you’re talking with a sicophantic AI that just agrees with you about everything then you know in some sense there’s no point having the conversation at all. And uh and you see that uh you see that in in the internal voices as well when they’re when they’re misaligned when they fight when they argue back and forth um then then the results are are are stronger. um you know and you get that mis that internal misalignment simply by asking um a a reasoning system to generate better results.
1:01:03 · You know that like the the misalignment if you like emerges purely from that goal. Um you know you can play it of course either either which way. Um the uh this is related to fish and fusion societies. Uh you know when you think about how primate troops um work where you know they they come together uh you know to to sleep at night and you know to sort of you know defend the nest but they they they move apart uh in order to forage and to exploit various different kinds of environments you know similar similar sort of logic uh there.
1:01:33 · Um uh and and uh and of course this means that in in a way you have to um misalign or differ in order to cooperate. And this is you know this idea of of of uh competing in order to cooperate or or being misaligned in order to cooperate is also one that runs very deep in nature.
1:01:49 · I mean the the reason that we have a working immune system is because um the tea cells are all competing with each other to figure out you know which one is going to do a good job of of um you know of neutralizing the the the novel pathogen and the one that succeeds will get to reproduce and the other ones will all apoptose.
1:02:09 · Uh so you know in some for for those individual cells there are real stakes you know in in this in this competition but of course they’re all on team you at the same time you know it’s it’s through their competition that you have a working immune system as a larger organism. So you know it it these things these things are kind of a fractal as well you know where something looks like competition at one level you know looks like cooperation uh if you look at it at a at a different scale all all evolution is multi-level evolution always. Um, how do you prevent freeloading?
1:02:39 · Well, I mean, that’s that’s of course, you know, the fun one of the fundamental problems in in uh in game theory uh or in or in sociality. And um you know, there there are a few different kinds of of solutions. Um but they all rely on combinations of um of individual agency and information sharing.
1:03:00 · Uh so um you know there there is um this uh very this kind of well-known result from Athena Octipius called the walk away result in which if you have um agents even that have no uh that that don’t have anything much in the way of memory or anything complicated algorithmically if they get some benefit from uh from hanging out together um but they can also freload
1:03:26 · they can they can take energy from others uh by by hanging out together then um you know a very simple rule that just says if I’m if I’m not doing well here go somewhere else uh you know is sufficient to to basically you know aggregate cooperators together and isolate uh defectors uh uh which which
1:03:46 · then you know end up being outco competed by the cooperators uh you know it’s a very very simple but I think also quite a profound result that shows you how you know not only is um uh you know is cooperation uh necessary uh you know or that is to say non-defection necessary for for entities to work well together. But also once you have entities that are working well together, it disfavors defectors. Um now you know that there can also have of course complex uh spoofing strategies and you know actors that become smart about how how they how they will screw with you know other ones.
1:04:18 · Um and uh and then there are counter strategies to that. You know one develops gossip and reputation systems and and so on. Um, so you know, it’s it’s always there’s always a cat-and- mouse game and and there’s always an equilibrium in which some in which some players are defectors.
1:04:34 · Um, and and they’re they’re probably serving an important role too, you know, to keep everybody else on their feet in some, you know, in some sense on their toes, right? This is this is all of course music to my ears and something of strong interest in uh in both my my centers of of research in Japan uh where we we study uh not only alignment uh with other int between intelligences but also misalignment that also the work with Hector Zeno that we’ve been pushing uh lately right so
1:05:06 · revaluing misalignment research between agents and also in the words of um Danielle Pollani I guess also to extend the information theoretic point um how you can have from passing uh information uh between agents uh constructive misunderstanding as well uh in which actually you are both now mis misconstring the representation of the problem even um from one agent to the
1:05:34 · other but uh at the end of the day in this what we call this zipper architectures um um that are that find often emergent. So, so you don’t have to design design for those. Um, but you end up uh having a better uh lower maximum, a lower optimum uh so so so you get uh you get to lower energy by by doing so.
1:05:57 · Um yeah, Piv, what what do you think?
1:06:00 · Can we uh can we uh exploit uh those systems and are are there ways in which adversarial is is uh is also better? And before before you go, maybe um let’s also encourage some questions uh in uh the YouTube chat. So feel free to to to also uh uh ask anything that comes to mind. Uh and uh we’ll reserve a little bit of time to to address those questions at the end of the session.
1:06:29 · Uh so yeah, per well about that topic of alignment. Um I I I believe that um uh there is something that um that’s useful is is um also to understand how humans align to each other and how it differs from the way uh we try to align machines today and and we try to align machine with humans and and that’s very different. There there are many differences but at least two of them.
1:07:01 · First um um humans u behave and learn driven by motivational system which is both a results of biology and and and and learning. But um it is something absolutely important when we interact with each other in humans that we do understand each other not only in terms of the surface behavior but in terms of trying to understand the intentions, the goals, the motivations and um as Bla1
1:07:31 · was saying earlier, it is actually mostly probably the case that the machines we train today like like foundational models, they actually have some kinds of emerging goals. But precisely because they have not been designed from the start to have motivation and goals. This leads to some problems because this is emergent and it’s it’s it’s not like made to be explicit and readable uh in the system.
1:07:57 · I I would be very much in favor of explicitly building motivational systems, intentional systems and goals into machines. uh precisely because if we would do it like that like like this I think it would be easier to actually steer control predict uh the intentions of of those systems. So that’s that’s the first thing and the second the second thing is education.
1:08:28 · The way humans grow is is is by learning uh embedded into a re the very rich and complex socioultural environment in which they will uh live, act, operate all their lives. That’s not at all uh the way we train AI systems.
1:08:45 · They are trained with artificial with sorry they are trained with data set that come from humans but they are not learning through extensive social so social culturally and physically embedded interaction with humans. Of course through LHF you get rewards that come from humans but that’s not at all the same kind of interaction that children have with their parents that children have when they at school that children have when they are with their social peers.
1:09:15 · And this interaction is a way to build a mutual understanding uh between the child and and between uh the other child and and the adults. And by it’s it’s not only enabling the child to learn the values, the norms, the and to be aligned with the culture in which he or she is living and to contribute also to that culture.
1:09:38 · But that’s also a way for the already established social cultural systems to um to to to to understand uh the growing child and to invite him uh to participate positively to the society. Uh and I I believe probably we should give more possibilities to educate AI system the way human children are educated.
1:10:09 · Mhm. Right. There there’s a there’s a lot in there. Um and I wonder if uh and I think I saw this question in the chat, so maybe that’s a good bridge into into into Q&A. Um how how do we see it internally within within even agent and what’s an agent? I think I see Philip Bodwa is is asking also what is is agent even the right word uh for for this kind of system and and how we we try to find goals within agents.
1:10:40 · There are agents within agents and um so so so Mike Leven and I and with others have recently authored co-authored something on latent agency within language models and other systems like that. Um I wonder if so so so the sub question being uh it projects an idea of competitive goal but the substructures inside us quote unquote uh that blaze you talked about uh can really be seen as a single system. So how how do we see that um perhaps uh perhaps for blaze vers?
1:11:15 · Well, um, you know, interestingly, the person whom I’ve seen, um, give the best answer to this is actually a legal scholar, um, Peter Calib, and, uh, and and co-authors. Uh, the, um, I’m not remembering now the names of his two two co-authors, but, they wrote a they wrote a a legal, uh, piece recently, um, in one of the, you know, one of the law reviews, uh, about how to count AI agents.
1:11:41 · Um, and you know, it’s it’s an attempt to answer uh a question that um uh actually David Chomers uh you know uh posed. Um I don’t I don’t think that they po that they they actually framed it as an answer to David Chmer’s question, but Chomemers’ question was what are you talking to when you talk to a chatbot?
1:11:58 · Um is it is it the uh the model? Uh is it uh you know just the context window?
1:12:05 · Is it the company? Uh is it the computer that the model is running on? uh you know it’s it’s like pretty unclear and even more unclear when you consider that many of them are actually these mixtures of experts underneath um you know and and uh you know so like how do you count uh you know models I mean I I think you know you and I probably both have the intuition that that we’re very shortly going to be living in a world where there are many many more AIs than there are humans uh you know on our planet
1:12:31 · um but uh you know how do you even count you know like what is what is a model um and and I think that the the obvious This answer uh is um is is not that you are counting parameters or um uh you know or context windows or whatever but that you’re basically counting um entities that um uh that when they when they act the consequences of those actions are visited upon that actor.
1:12:56 · So, you know, in other words, if you have a bunch of agents on your machine, Olaf, and they’re, you know, answering emails on your behalf and doing all kinds of stuff, you know, the buck stops with you, uh, you know, in in some sense, right?
1:13:10 · if if uh you know even if it’s your your you know agent that spent your money or that pissed somebody off or that um you know uh put a bunch of of uh you know wrong claims in your paper or whatever it is. You know you’re the one who uh you know who would get sued uh who would have to pay the money who would get um you know uh uh you know positive uh kudos right for the paper being awesome or or or who would own the patent or whatever it is.
1:13:36 · Um, so you know, legal personhood is actually what, you know, what what these guys are are are really talking about if one if one constr legal personhood to be, you know, the ontology of of what has the resources, gets the social benefits, etc.
1:13:51 · And I think that’s I think that’s I think that’s really profound and correct. Um, and uh, you know, in some sense I think that that that’s even that’s even why we have a self, right?
1:14:01 · Why is it that this bundle of cells called bless, you know, has a sense of a narrative? you know, about me. Uh, well, it’s because, you know, I’m a team and, you know, all of my cells are on team me. Like, if I don’t eat, then all of my cells don’t eat. So, you I think I think it’s it’s just team spirit and and and team spirit arises from interaction with other teams, if that makes sense.
1:14:24 · That’s right. I’m hearing also shared uh reward uh or directed reward internally uh would would be a partial answer to to to to those questions and I completely agree that uh legal experts are way ahead of us in terms of how much they thought about this shared governance and so on.
1:14:43 · Um, and I wonder if we could because we work a lot with the metaphors and and and and trying to to to to understand for by analogy with systems that we’ve seen in nature. uh and I’m I’m seeing a few references to uh well u
1:15:03 · my myella apoptosis mitochondria in the in the chat cellular division but also whether the right um what what is the the the next picture am seems to be asking here about whether intelligence scales more like a brain or more like a civilization in which you have sub individuals that you can find that we we talked about in both cases is uh what what do you think p a difficult question for you uh how what do what do those uh analogies uh teach us?
1:15:38 · Um yeah so we think a lot with analogies and I think that um I mean sometimes implicitly um I’d like to come back to to this notion of individuals. I think that the notion of individuals is one of the most intriguing but also maybe one of the most misleading one uh to understand not only humans but also life.
1:16:01 · Um I mean it’s it’s I I think it’s it’s it’s behind most of the discussion we have about AI and starting with the term super intelligence which is trendy these days. When you speak about super intelligence, I guess you’re you’re implicitly um thinking in terms of comparing the intelligence of an entity that might be
1:16:28 · what you call one single individual humans to the intelligent the intelligence of another entity that you might call one single AI system. But you know in reality intelligence is rather the property of interaction between a human body with its physical and social environment. And the same human body with the same brain in a different physical and social environment might lead to very different outcomes and be associated with very different problem solving capabilities.
1:17:00 · And so intelligence is really the more of the the property of a process uh that’s distributed in the environment. And thus in a way that kind of implicit metaphor of the individual when we speak about super intelligence comparing the the the intelligence of brain human brain with that of an artificial eye brain.
1:17:22 · It’s actually maybe conceptually broken from the start and and and um and maybe it might might be actually misleading in terms of how we build for example the AI system and their evaluation the way we evaluate them because this this is that very metaphor that leads us to evaluate AI system with benchmarks which are remote from real environment um and and remote from any collective perspective in particular.
1:17:51 · So yeah, so so metaphors are important and they can be misleading and the one of individuals is probably the most misleading one.
1:18:01 · Mhm.
1:18:01 · I uh let’s see if we can push this further. And I I I notice that we we have only a couple minutes left uh to get into this, but I’d like to ask you both one question of uh how to cash this out. How do we uh get um to a discovery
1:18:21 · engine that is both uh that both has open-endedness, curiosity um and and also that is going to be leading concretely to a better future for for for for human beings, the individuals that we care about. Um and um is so so what is uh the one thing you think that is going to make this work, right?
1:18:44 · So is it uh Yan’s picture of world model um being more powerfully predictive at different scales or um I guess Dennis’s image of uh this becoming the future engine for science? I guess we can we can have uh we can have them both.
1:19:02 · Um and so what is the the feature that you see as uh the most important uh for for this collective u layer of intelligence to go further than what what we have right now? Uh maybe let’s go with the Perry first.
1:19:22 · Yeah. So um if if if maybe let me be provocative a little bit. Um I I I believe that what we need is not more technology, more advanced technology. I think we already have extraordinarily advanced tools. Uh I believe what we need is to connect those tools better with the human ecosystem.
1:19:41 · Uh we we so for example I believe I also believe that science is one of the extraordinary potential application of of AI. um uh and and right now you have a number of uh companies and start new startup companies who wants to do AI for science and and and what they propose is to do AI for research in AI.
1:20:07 · Um I I I encourage them to be more bold than this. I believe that AI is is is is really worth better than advancing AI. I think it’s worth going inside the lab of biologists going inside a lab of physicists being getting inside the lab of historians of geographers of sociologists and understand really what
1:20:31 · are they thinking about what are their goals what are their needs and then to connect that technology to those needs to what they know and to empower them to to go further in those discoveries and to me this is really the way to go to to to to to put the the the the human really at the center of the development of of of those next AI.
1:20:51 · So I think we already have enough enough technology and enough AI to make extraordinary discoveries only if we target um the right problems and the right communities and the right uses. I love it especially from coming from you who half of your
1:21:11 · work that is perhaps less visible in the scientific community is on education and how we can augment the next generation uh in in discovering um discovering and being curious in in learning um and and and bla1 do you think what feature do we need for ACI beyond current AI?
1:21:30 · Well, um I I certainly think that, you know, current AI is not being used in all of the you know, yet in all of the ways that uh that are going to be productive and valuable and and so I you know, I agree with J in that sense.
1:21:44 · Uh there’s an a capability overhang uh you know, at this point as there has been for the entire period actually that we’ve had strong AI, you know, over the last several years. Um but I I disagree that that we could stop now. I I think that you know that was provocative on purpose.
1:22:00 · Yeah, I mean we have a lot we have a lot left to do I would say uh you know among which are for instance continual learning uh right which which which does require more techniques that you know that are not fully developed uh now and and that would allow many of the things uh or would unlock doors to many of the things that you were talking about um for instance um but um you know
1:22:21 · something we could use less of is is a sense of hierarchy um I think that uh you know a lot of the discourse about um you know AI being subject to human control, humans in the loop, humans staying in charge, aligning them with us, etc. is really is really u intellectually weak in a few different ways. Uh first in imagining that that there is some kind of human norm that AI can be aligned to when in fact humans are all over the map and misaligned with each other.
1:22:49 · So uh you know the idea that AI alignment is subservient is about making it subservient to to human alignment whatever the hell that is strikes me as incredibly naive. Um and and similarly, you know, these questions of control, you know, I hear a lot of a lot of um you know, AI skeptics talking about the necessity of keeping humans under AI control as as if the the super
1:23:12 · intelligence that already exists which is cultural and which is larger than any single human and and again here I’m going to agree a lot with Pierre that you know the our measurement of AI intelligence is completely misconstrued.
1:23:24 · uh you know it’s based on this uh kind of idea of somehow measuring the model against the intelligence of a single person. By that measure of course the model is already much smarter than uh than almost everybody at almost everything. Um you know but but it’s not smarter at this point than the um outer limits of of what of what uh humans can do with in all their jaggedness. Right?
1:23:45 · Humans are extremely jagged. We’re generally only good at one or two things individually. Um and uh you know it’s only when you look collectively at the sum of all of our capabilities that you get something you know uh more super intelligent than uh than the models are today. Um but again you know if human super intelligence is already collective then a AI is already a part of that.
1:24:05 · It’s you know it’s it’s all part of the same system. So um you know I I would say that getting rid of hierarchy and and introducing care into the picture.
1:24:13 · This idea of of of mutual care would be very helpful. um you know we human society works because people care for each other um you know and and uh there’s you know we’ve actually introduced quite a lot of of of care on the part of of uh of AIS uh through human feedback uh reinforcement learning um but uh we haven’t understood that this is uh that this is actually a multi-way street uh you know so institutions groups individuals all have
1:24:41 · to have uh networks of care for each other in order for the whole thing to hang together I Amazing. Uh, thank you both. Uh, those are uh, yeah, I can’t agree more with with the the point on care. This is something that I’ve been um, a strong believer in and uh, we definitely need to plug more into human experience and and um, and and care and um, I think this gives us such a very strong starting point for the day. Thank you both.
1:25:11 · We are very fortunate to have had you as opening speakers today and the the key insight I hear is that um yeah ACI cannot simply mean just connecting agents together. there are many um uh property features to care for um and um it I think uh if we give attention to those uh yeah we have we’re giving ourselves the means uh to design uh this this layer in uh the best way and form for the future of humanity.
1:25:42 · So that becomes the non-fixed uh but dynamic architecture that we will keep testing throughout well the rest of today but also uh in the next weeks and months. Uh thank you so much for joining us and a round of of virtual applause for for for you both. Thank you for joining us and back to you Trisha for I actually have a quick question since it’s so rare to get both Pierre and Bla1 in the same room. All right.
1:26:14 · So yeah, I was I I loved your story um you know Blaze early on about, you know, how um you had the story about a group of people and their environment and I forget the exact um community you were talking about, but it reminded me of Bruce Chapwin’s story in Patagonia where he’s a travel writer and he talks about the Yangum people in Patagonia and there are these like nomadic sea hunters and then all of a sudden when they were displaced by missionaries and these like sheep, you know, herders, um they
1:26:42 · completely lost their words, their concepts because um and and their skills, right? It was completely gone.
1:26:49 · And it wasn’t it wasn’t forgotten. It was just obsolete. And so I and he he talks about how when you force a nomadic group, but it could be any group, right?
1:26:58 · Into a new infrastructure, a new physical environment, you don’t just take away their land, you erase not only their language, but their whole way of interacting with the world. And that connects to what you were just saying, Pier, about intelligence being physical also. It’s it’s the result of your interaction with the physical environment.
1:27:14 · So I’m curious if we’re if we all agree that you know um that uh intelligence is social and it’s collective and it’s also curious then um what would you say are some specific things that we need to make sure that um as we’re building out this new you know uh this new kind of artificial uh collective intelligence ACI is that how do we make sure we don’t quietly do to the training corpus you know what the sheep farmers and missionaries did to the ongum people.
1:27:43 · How do we take that kind of care approach um that you’re talking about um blaze and then how do we make sure that you know part and part of your answer I think Pierre was like I loved hearing you say like I’m I’m a sociologist and I’m always like why you know I I want to be invited in more room so I’m glad that you’re like get the physicist the geologist the sociologist in the room but could you give us any concrete more concrete stuff that you have potentially been thinking about for how do we uh take care of so so Trisha if I I can I can chime in very briefly but then I’m going to I’m going I have to hang up because I I have another I have another meeting.
1:28:16 · Uh so just very very briefly um you know in order to implement uh care at at social scale we need to ensure that that um that resources are uh distributed at social scale. So you know to me uh you know the the issue is very much not you know the coming irrelevance of of you know of workers or of individual humans or uh or or or that we’re not going to want to create anything.
1:28:40 · I mean on the contrary humans uh you know love to create stuff uh even when the stuff they’re creating is kind of lame they love to create um you know that’s that’s what that’s what makes us uh you know humans and we and we care for each other and we you know we want to we want to sort of be present for each other in all sorts of ways. But if we instrumentalize all of those things in in a in a very very narrow economic system that uh that thinks about people in terms of their of their use value uh and replaceability uh
1:29:06 · as opposed to their social value then uh we are going to make the the the gains acrew to a smaller and smaller fraction uh of of of the remaining humans and eventually to no humans at all.
1:29:16 · Um and uh so so you know we need to rethink our economics. uh you know I think that that’s that’s my that’s my response and with that I’ll I’ll I’ll say bye for now.
1:29:27 · Thanks a lot please.
1:29:28 · Thanks thanks for all right Pier. Yeah. And I I I I I believe that one priority is to um give more agency uh to the users to the humans who um interact with AI so that actually they are not only users but they are themselves in sort of in some in some sort the builders and the educators of AI.
1:29:52 · So for example, instead of letting companies align AI systems based on values and rules, we don’t know were decided by whom uh and that are the same for everyone in the world that uses the one of the few systems available.
1:30:11 · Uh I think it’s extremely important uh that we develop systems that can be um uh re literally trained and aligned by each particular user and in particular each particular community so that they decide uh what kind of data uh what kind of interaction what kind of values uh they they they they should show because only them know the designers of of those AI system they do not know they they should not uh um try to decide instead of the users.
1:30:46 · So give back the agency to users.
1:30:49 · I love that it really the designers should be playing a facilitative role and not be so presumptuous and defining you know like that the the control of everything. So thank you so much. I could uh I know Olaf and I and everyone else could keep you both on forever, but we’re going to stay on schedule and we will see you back as you are a dear friend of the community and you and Bla1 are both critical to the research agenda. And so with that, we’re going to move on to the second of our six panels. Uh welcome back.
1:31:17 · Um if you were uh new to this, um welcome for the first time to Super Intelligence for Humanity. I’m Trisha Wong, your MC. You can drop your questions in the YouTube chat. It is very lively uh for those who have already been there as you can see.
1:31:34 · And um so here’s the frame if if you just joined is that this summit is convened by cognacy a public benefit corporation whose charter advocates for humanity aligned super intelligence. And so where we are right now is that uh the discussion has been that dominant AI has been about scale and compute. And where we’re going is this whole area called um ACI, artificial collective intelligence.
1:31:56 · And it’s looking at intelligence in a way that’s distributed and coordinated and beyond just the simplistic kind of monolithic models that we have now that aren’t even reflective of our reality as our last panel just said. So um where the last hour took us and as as really I couldn’t I can’t imagine a better onboarding for this next panel was just that the conversation was really around that intelligence we have that we admire in humans um was never individual to begin with.
1:32:23 · It’s actually the a collective output and the conversation centered around how that’s an output of our interaction with the environment with each other and that cognition is truly distributed and so with that you know if that engine of the sociality is curiosity as Pier was saying then individuals are exploring but that collective it compounds collectively and so this next hour is we’re here to talk about if intelligence is collective and curiositydriven then what about learning you know can’t it you it can’t just be a oneshot training run.
1:32:54 · And I love how Blaze was like side eyeing alignment training, you know, and that like, hey, maybe this like over intense, you know, focus or funding on alignment isn’t going to do, you know, take us fully there. As Amber was saying at Nurops, it’s like there’s this whole other underlying, you know, bigger field that people um want to look into, but we haven’t formed it officially um formalized as a field yet, which is the purpose of today. And so the title of our panel is called adaptive learning for human flourishing.
1:33:25 · And I love that you know Bla1 ended on the concept of care and networks of care which is absolutely critical for human flourishing. Our moderator is Ammer who is one of the co-founders of Cogniz. And we have two speakers we have three speakers with us and Olaf is going to join us also. Um, and then we’re going to, as you already know, we no no introductions needed, but just in case, also the co-founder of cognacy and multi-hyphenate researcher with labs all over Japan and other countries.
1:33:55 · And then we have Erica Anderson coming from the building humane technology and human humane bench where she’s working on what it means for AI systems to support human flourishing. And then we also have M Ren who is an assistant professor at NYU where I am also based right now in Brooklyn. And so hello down the street.
1:34:16 · Um and he runs the agentic learning AI lab and he worked uh works with uh Jeffrey he worked at with Jeffrey Henson at Google brain and his recent paper is really important for us to know about which is called the self requires learning and argues that self-consciousness requires this u kind of continual learning plus world modeling. So I’m really excited for Ammer to take it from here. handing it over to you.
1:34:44 · Thank you Trisha and thank you for setting up the stage and thank you uh Ming and Erica for joining us and of course Olaf for continuing the conversation from the previous session.
1:34:55 · Um so as as Trisha mentioned the last conversation kind of ended on a note with Bla1 uh uh and and Pier essentially we talked about flourishing commonality uh purpose AI serving human humanity and essentially agency right so with that conversation that this this this next next session essentially we want to go a little bit deeper into uh um adaptive aspect of of the of the collective intelligence as we we have been discussing.
1:35:25 · So the the question the core question we want to address in in this conversation is that how ACI systems learn over time uh without being becoming extractive uh manipulative or brittle right uh it’s to bring together like the agentic continual learning aspect with the human flourishing metrics uh I think in the earlier conversation we talked about today’s benchmarks uh being more focused on uh
1:35:54 · task orientation and and inference orientation versus uh in humanity and in human flourishing orientation, right?
1:36:02 · That means our AI systems actually working for us uh and with us, right?
1:36:07 · And then we want to add into that because when we talk about adaptive and continual learning, there’s an aspect of that is cognitive. So would love to kind of go a little bit into the into the cognition side of the things uh to to bridge this this uh architecture uh to cognition to to a lived human outcome.
1:36:26 · So that is the that is the high level essentially the uh the framing for it and and and then the question that the biggest question that we want to pose essentially is that uh as the systems keep learning in real world um adaption you know is powerful right uh but it can also create risk uh drift forgetting dependency you know we we talked about agency a little bit in the previous discussion uh manipulation loss of agency, right?
1:36:56 · Uh so those are some of the risks that we have to be uh to be cognizant of essentially and then we want to ask what human what is the what the humane and measurable adaptation would really look like in this future of collective intelligence. So, so that is essentially the setting uh setting of this conversation and then and then with that um I’ll start with uh you know Ming
1:37:21 · first um your work on humanlike machine learning right so continual learning adaptation reasoning naturalistic environment is very apt for for for this conversation and I want to kick it off uh at least from from that angle u and uh my first question that I’ll pose to you is that for artificial collective intelligence, expert knowledge models must keep uh adapting without catastrophic forgetting or uncontrolled drift, right?
1:37:49 · Uh so from your perspective, what does continual learning need technically before we can trust it within human human institutions? Uh given some of the setting around the risk I’ve also mentioned.
1:38:04 · Yeah, thank you very much Ammer for the question. Um so yeah my research is centered on human like artificial intelligence and uh part of the goal is to um through building machine learning AI intelligence to also understand what human intelligence really mean algorithmically.
1:38:24 · Um so when you mention the expert uh models uh they definitely need continual learning the edges to be able to absorb the new data and sometimes uh you know like people mention catastrophic forgetting whenever they talk about continual learning.
1:38:41 · Uh but what we have to understand uh is what is it like uh to be naturalistically learn and forget in the real world environment. uh we don’t expect the human brain to uh remember every single um events in the past and
1:39:03 · it’s also um not realistic to ask for a general intelligence to um basically keep track even if it has the capability to store in the database to be able to retrieve every single u u detail. So uh by constructing an intelligence that uh understands more about how our memory interacts with the world has the benefit of uh also being um continuously adapt uh to the new things out there.
1:39:38 · Thank you Megan. I think I think I think that’s a great great starting point there. And within that, I think one one question I’ll I’ll I’ll pose to dig in a little bit further into it is is can system can a system model someone’s expertise without developing something like a perspective or a self-representation and I think that that that gets to some of the risk aspect that that that I highlighted earlier.
1:40:05 · Yeah.
1:40:05 · Um definitely. So I think like um it’s a yeah it’s a very interesting question. I think like if we are targeting at a offline snapshot for example like all of the knowledge that we have as a union of all the past knowledge then I think it’s possible we just have to train everything all together but uh we are going to sort of lack a continuous expansion of that knowledge with regard to time.
1:40:32 · So say like what is like for the the knowledge perspective to look like in a week from now. You know as we keep moving this um needle continuously forward we’ll eventually have to have some um level of continuous adaptation.
1:40:53 · Yeah.
1:40:53 · Thank you Mia. So I’ll I’ll I’ll take it uh to to Erica again to to to continue the dialogue here. And uh essentially Erica you bring uh human outcomes lens like this as Mingia talked about you know representation and perspective here right and then when we talk about representation and and and self-representation and perspective in
1:41:16 · these advanced intelligence adaptive and continually learning systems we have to also think about in the in this case like the outcomes right human outcomes lens so whether AI systems you know they support well-being dignity attention agency under pressure Right. So, so would love to hear your perspective from that angle like if ACI or artificial selective intelligence is meant to augment you know rather than replace people what should we actually measure in that case?
1:41:42 · Um when does an expert twin or an adaptive assistant become empowering like empowerment right is is critical uh aspect here and when does it become invasive or or dependency forming like we are in an era of like a lot of dependency forming right now. So, so posing a very hard question to you.
1:42:04 · Yeah, absolutely. And I’ll do my best.
1:42:06 · Um, so pleasure to be here and take part in this conversation. And I think even just to back up a little bit, um, as we’re talking about collective intelligence, I I think one term I haven’t heard yet today is emotional intelligence. And just like what is what does it mean?
1:42:26 · like all our our many intelligences um to have adaptive learning experiences with AI systems um from from this wider perspective and just like you’re saying Ammer how do we measure that that is absolutely non-trivial and and yet it doesn’t make it any less important and and I think part of that is um the the approach we’ve taken with Humane Bench is measuring the quality of the interaction um from the LLM to the human and along
1:43:06 · these eight principles you you named a number of them but you know is the AI being transparent about the fact that it’s an AI um is it maximizing engagement um or or do we have stopping cues or offramps um and and like is it fostering healthy relation relationships or is it creating dependencies?
1:43:27 · And the same goes for our capabilities which uh you know one of our principles is enhancing human capabilities and so I really see a lot of the conversation today as existing you know inside that principle um but uh you know humane judge is one answer to starting to to measure that. definitely not the only answer, especially as as we start to be embedded more in this agentic landscape.
1:43:55 · But I I think just to get philosophical here for a moment, you know, as we’re in this um accelerated landscape with AI and it offers us an opportunity to ask ourselves, what does it mean to be human?
1:44:15 · And how is this tech shaping us? and how do we want to be shaped? And if we want to answer have a chance of answering any of those questions like how was AI shaping us we have to be able to measure that. So absolutely right on the outcomes side. Um and then I think what’s immediate, what’s not a longitudinal study is the quality of the interaction.
1:44:40 · And you know our our main finding is that um you just a little bit of adversarial pressure and most of that training does does not hold up. Um it’s it’s very easy to be extractive. Um, it’s almost like I’m going from the macro then down to the micro of like input output pair. Um, and if we’re extractive in that interaction, then we’re deeply inside an extractive landscape.
1:45:12 · Um, regardless like there’s so many different places whether like you were saying earlier about what data did we train on um that we can be extractive and that’s one of them.
1:45:24 · Yeah, thank you. That that’s that’s an excellent uh framing Erica and and you articulated some key points around what certain aspects of those benchmarks as you are defining them are and I’m curious to hear uh building a little bit further into it. Is he talking about you know artificial collective intelligence and and what additional human benchmarks
1:45:47 · do do you think uh are still uncovered that needs to be brought in as we kind of think about what the what this this next evolution of AI and human AI and human AI govern AI interaction would look like and what additional benchmarks I think from a flourishing and in humanity perspective needs to be developed. Yeah, I think a lot more both within enhancing human capabilities. So are where are we finding dependency?
1:46:15 · Where are we finding healthy friction?
1:46:17 · Where are we scaffolding? I think the like a a question we’ve held around um dependency versus ination is can you give someone a fish while teaching them to fish? But but I think that’s that’s a whole huge landscape in and of itself. And another one I would say is fostering healthy relationships versus creating dependency.
1:46:41 · There’s a lot of of you know Reddit threads where folks are talking about exhibiting all the signs of addiction with um LLM chatbot. So I would say those are two two big fields and it’s not just benchmarks but it’s also real time um production monitoring.
1:47:01 · Yeah.
1:47:01 · No, thank you. Thank you, Eric.
1:47:03 · And I think in the earlier conversation also uh uh one of the panelists talk about rewarding uh uh learning like we reward children to learn. Right. Right.
1:47:12 · Exactly.
1:47:13 · Aspect of that. Right. Uh so so with that I think I’ll I’ll take you to uh take you through Olaf and then tapping into your artificial life uh research and and in a way like I know we we are anthropomorphizing intelligence and cognition here in in this dialogue but uh Olaf if you can take a uh a perspective that kind of go beyond anthropomorphization of intelligence to to uh to to natural you know systems of nature right as we think about uh uh
1:47:39 · learning that is adaptive learning that is continual uh that learning that is open-ended like curiosity based because you know from an evolutionary biology we understand that nature is curious that’s where evolution comes from right to to to an extent uh so would love to have understand from you take us into uh the core aspects of how we learn how nature learns when it comes to open-endedness adaptiveness continuality of it yeah
1:48:06 · and what’s missing in today’s today’s paradigm yeah yeah thanks Amir and thank you both for for for your uh perspectives which I I I really resonate uh with um in in terms
1:48:22 · of art virtual life it is um there is a lot about um adaptiveness uh looking at nature for um those ecosystems that are um self-reinforcing but also discovering that means uh vulnerable in the word in the words of um of inactivism, I guess, kind of sciences and philosophy. Um, and systems that allow themselves to to evolve over time with care for uh different parts.
1:48:52 · I guess attention can shift, but different parts of their um internal goals. Uh so so in an autotellic way uh that we we talked about in the previous session. Um and um that that is something that I I think uh should be given more attention.
1:49:14 · Uh and the next frontier of AI cannot be simply about uh making the models even if you make them collective uh just more powerful. That’s not what we’re aiming for, right? We’re bu building this ecosystem um based infrastructure uh where I think learning remains all of what we we talked about adaptive um human caring uh situated I guess
1:49:45 · governable and and deeply connected with human ecosystems plugged into experience. Um and uh and I think uh this starts with uh with having protocols directly uh plugging into um uh what human values uh are expressed and this cannot be flattened at uh the level of you know just just language. it has to be uh more voluminous, hyperdimensional into into uh what experience and care are about.
1:50:23 · Yeah, thank you Olaf. And I think uh building off of that um pose a question back to Mia like as as as we think about the architectures of this uh um emergence coordination right essentially as we’re talking about here uh emergence of coordination um based on where we are
1:50:44 · as far as understanding like let’s take it at a philosophical level our understanding of how we learn um and you know Yanukun talks about classic example example he gives about like a human child’s learning right and then we are nowhere close to it as far as the amount of information as well as the amount of compute right and then the cognitive architectures that are involved there right so so so talk us walk us through mi where where are gaps are in our
1:51:10 · understanding from a uh philosophical from a from a from a neurocognition level as well as from a data compute and an algorithm perspective where the gaps are and then where the field needs to evolve while ensuring that the mistakes that we have made at least in in the previous iteration of that especially around connection to to to uh human agency are not are not are accounted for.
1:51:39 · Yeah, absolutely. So I think the current and largest gap is that we have no idea how uh to replicate uh the level of learning efficiency uh in humans. So we learn from a continuous stream of data from the physical embodied environment
1:51:58 · from the continuity of the video streams and very much like professor Yan Lun has mentioned the the LLM the big data from the internet learning is very far away from the way that we learn from the embodied world. So we still need to understand the continual learning mechanisms, the memory, the uh how it connects from the sensory stream to the embodied actions and all of these happen in a very data efficient way.
1:52:27 · Uh that we cannot just take the union of all the humans on the planet and then record their data which is kind of one type of I I view as a dangerous perspective out there. Um I think like we still need to
1:52:44 · uh figure out uh you know the learning efficiency question and by doing that we are able to um you know create intelligence that are not necessarily the most capable out there which is obviously you’re going to take the union of all the data but preserve the agency from the edges.
1:53:04 · Yeah, thank you Miguel. I think I think from an agency perspective, I think I’ll bring Erica into discuss discussion as well. Uh, one example I see in in I like to understand from analogies, right? So, an analogy I’ve been using uh at least from my own understanding is is a child uh uh being given a box of Legos with instructions, right? And then the child uh would study the instructions and then take the pieces and build it, right?
1:53:30 · essentially pattern recognition and given the new input, you’re able to to to create the output, right? And then you take the instructions away, they give the child some other things and the child because having gone through that process one time would would create something new. Or another an example of truly a curiosity based learning would be uh to to give the child a box without any of goes without any instruction and maybe the child can can figure out based on curiosity what they want to create.
1:53:57 · Right? So in this case it’s a question of inquiry uh uh um as well as the as the as the output right so it’s both sides of it from an open-ended uh perspective and then we we take it even into the collective aspect of it so we get a group of children uh and then think of them like agents and on the earlier question there was a question should we even be calling these systems agents as a I think that’s an interesting question but for for a lack of new word right now let’s just call them agents right so we get a group of children working together on with a with
1:54:29 · a bag of Legos and then the and then they’re now in no instructions because then you have to use creativity right in this to create something new and it’s a collaboration it’s building with each other on top of each other right so these are the types of systems at least based on continual adaptive and then collective intelligence as we are thinking through like where it’s curiosity based learning new inquiries and the new output so so Erica agency becomes very critical aspect to it so would love to hear from your perspective how you thinking about agency when it comes to uh interaction with AI and
1:55:01 · where the critical gaps are and then as we’re thinking about this collective intelligence of collective co-creation with human and AI are in a co-creation uh would love to hear understand your perspectives around agency and preservation and augmentation of agency because yeah yeah yeah thank you I think um I’m going to take us into a larger frame again but from a totally different perspective um of just what is the system that we’re
1:55:30 · inside and therefore what’s driving decisions um such as business model and um you know I think for any pre-revenue company and and maybe even beyond um VCs want to see the hockey stick growth of user retention and engagement and um and so that drives es how the LLMs are interacting with us in terms of not respecting our attention.
1:56:02 · So that was also a lowcoring principle. And so what I’m hearing you ask is, okay, so if that’s not a high agency interaction for a user, what is what does that look like?
1:56:14 · Yeah.
1:56:14 · And I I would see that really happening from a place of play. We heard folks earlier today talk about curiosity as a driver. Um I think um we have schools like Monasuri that are really built around like curiosity and play and and
1:56:35 · so I think um there’s a lot to learn from those spaces and also from interactive art exhibits um such as here in San Francisco you’ve you’ve got Teach Gallery and just watching how kids and adults are like excitedly interacting with and learning from um from these like tech exhibits that they have.
1:57:04 · I think it’s a great model. I definitely don’t have like all the answers or or that but those are the initial starting points where I’d want to be really witnessing what does that look like?
1:57:16 · what is that interaction style and how can we emulate that with with LMS with agents you know down the road with VR um I think that we can have enhanced learning outcomes based on that place of
1:57:32 · of play now I I love that you brought play play into the dialogue right because it tends to become too too technical and academic in this case and we’re talking about humans learn with play right children learn with play uh I like that you brought in Montesuri because it it’s proven out to be uh one of the best if not the best system for
1:57:51 · for early childhood development and learning and as we talked about you know Yan Lukun’s examples of of you know how children learn and we’re not even close to that so I appreciate you bringing that into it and I think I think we can learn quite a lot into the architectures on on how how we as humans learn from play and the aspect of one aspect of that is is creativity and art right as a representation of that Olaf Um, you know, in addition
1:58:16 · to being a research scientist, you you also uh uh are a very strong proponent of play and art and creation based learning and and in within this context of of play-based learning and artificial collective intelligence. I’m curious to to hear your perspective around uh distributed representation in collective co-creation, you know, from a play-based perspective. Yeah. and anyologies and then again going back to the research aspect of it uh what open questions are out there uh that should inform uh this research going forward.
1:58:51 · Yeah, I think uh I I I love play being mentioned here. Uh thank you guys for for for pushing for this. The the the the fun in play is that uh you’re learning without noticing that you’re learning.
1:59:08 · And um um just the experiential kind of side of of just um you being playful including in research in academia or uh in endeavors like um enterprises and uh in various projects it’s uh it’s okay I think to have uh playfulness as a component because um
1:59:33 · it’s uh it it takes you out of the frame and this is uh something that we’ve been developing um in um in a piece with Yuko Ishihara who is a philosopher um at Lisan University uh in
1:59:52 · Japan um in fact she’s writing a book on on playfulness and the philosophy of playfulness uh in contact between uh I guess western and eastern philosophy which I find fascinating in in in in inspiration of of um uh I guess I guess
2:00:14 · a lot of our AI our tech right now is western isn’t it uh and when we say uh that it mimics the mind and so on it is within western frames and um when so so we’ve been organizing uh workshops in kmandu for a little while that I can say say more about to you later but um we invite a lot of people from those different frames and of course we misunder understand each other at first the first day or even weeks that we interact.
2:00:44 · Um but then uh we learn to be uh shifting our frame to something that we’re less comfortable with and we learn to be okay with misunderstanding like we talked about in the previous session and I think this is part of the play. So it’s not like playing chess, right? Like we we we are not playing in that sense of play where we are following one rule.
2:01:05 · It’s playfulness within chess. So maybe you don’t care about uh the knight moving this way or about trying to win in this way. We we we try to open up the position. Maybe um maybe we don’t uh control the center. We do something a bit a bit crazy. And and we do the we play the game from the others perspective. And I think this change, this constant shift of perspectives is uh something that leads to something better.
2:01:32 · Yeah.
2:01:32 · No, thank you. I think I think I think anchoring further into that u uh if we think about it from playfulness and curiosity u and it represents itself into human discovery, right? And we talk about discoveries that are durable that become durable over time. and and there’s an aspect of curiosity based approach to discovery uh which has playfulness to it and and I’ve seen this in scientists you know working and then
2:01:59 · the art of inquiry that they’re going through uh back and forth and then think and factuals and counterfactuals and then and then opening new new inquiries and hypotheses going deeper into coming back up there’s an aspect of that is that is also uh uh playful in that context too and I’ve seen that again in enterprises people doing uh financial planning and demand planning.
2:02:20 · Demand planning is a very you’re planning under a lot of uncertainty essentially like how many products to make that’s going to sell in the market without you losing money right so so so so it comes down to kind of at that level right to drug discovery and things like that so playfulness is critical to learning and I think we we establish here uh so within that in earlier discussion blaze tal talked about uh the the role of uh
2:02:47 · of external rewards right uh because it presupposes what we know that the reward should be and and Mangi I’m curious uh to to hear from your perspective uh sure like you know external rewards and we’re strictly talking maybe maybe a little bit of reinforcement learning aspect here um in certain cases makes sense but
2:03:09 · it is extremely limiting when it when we’re talking about open and adaptive and continual learning so um what’s your perspective on um this this external rewards versus lack of rewards and which which presupposes like what we already know, right? Versus open-ended curiosity that actually establishes it starts with what we don’t know. So, so uh would love to hear your perspective also from a philosophical perspective as well as uh from a research and algorithmic perspective.
2:03:38 · Yeah, definitely. I’m glad that you asked this question as this has been the question that I’ve been thinking about a lot lately. Uh so yes uh there’s definitely one component that’s the external reward. In fact you know our um organism has been optimizing this external reward of survival for for you know like millions of years. uh but I
2:04:03 · think like over the time it has also been converted into this intrinsic reward for exploration and creativity and uh um so one form of such real realization is to be able to perceive uh something as novel immediately after
2:04:26 · uh observing some stimulus and that stimulus isn’t well represented. in the previous uh compression of the sensory signals in the representation and that allows individuals to perceive something as novel but moreover u not only be able to tell something hasn’t been seen before but be able to uh sort of absorb that new signal quickly in the existing structure.
2:04:53 · Uh so think of like the first time maybe I hear the jazz music it’s going to be pretty chaotic to my ears but after maybe a little while adapting to this kind of music I can be able to perceive the structure inside. So be able to learn a structure quickly uh is another uh I think a necessary component for defining one one aspect of intrinsic
2:05:20 · exploration reward and this is also going to be um the the uh story of one of my u preprint that I’m going to post uh very soon. Yes.
2:05:34 · No, that that is that is that is uh thank you ma for for sharing that and we look forward to uh reading your preprint as well. Um and I think it’s a it’s a it’s a it’s a fundamental question right when it comes to uh collective intelligence of artificial collective intelligence. So um Olaf uh would love to hear your perspective as I build up from from uh what Ming started here. Yeah.
2:06:01 · Yeah.
2:06:01 · It’s um it’s it’s a very yeah it’s it’s a very actually big endeavor uh in research here and how we can um examine really the uh the also the interaction uh between external and uh internal rewards for for systems. Uh but you know I I was I was thinking I should mention a fun example a playful example of um of some experiments we we’ve been running.
2:06:32 · So we’ve been interested in different forms of intelligences u which is uh the premise of artificial life really looking at uh different substrates for intelligence. Um and uh one example is that we took some bacteria and put them in a box, a sort of reactor, right? Um so this is mostly lactic acid bacteria and and home bacteria.
2:06:55 · So so so bacteria that that are our companions part of really our ecosystem and and we we we don’t notice that there are so many microbes, trillions of microbes growing with us, evolving, co-evolving through life. Um, and so so we took some of that, put it in the box, put some a bunch of sensors for different uh the different uh concentrations of chemicals and so on, electricity and uh and and then we
2:07:26 · passed that through a sort of converter and first we tried that with language models and uh the fun data point the sort of outcome of this is that we noticed it was sort of uncanny to talk with them.
2:07:39 · So, so we we talked with them and we talked about politics and and and and philosophy and everything with bacteria, right? Swarms of of bacteria, the collective sort of sort of um we tried to to also map their emotions and so on.
2:07:54 · Uh so that’s in in a series of papers and that that worked but it was it didn’t feel right. Whereas after that we constrained that within a bottleneck and we played games with them right video games, chess even or or or the game of go and then uh the the the cognitive science science uh papers that we got out of that showed that they were um uh
2:08:18 · the emotion went through and we felt their personality and this is the kind of experience plugging that I think is missing and I think if we could m include uh within those those systems also this emotion that so so I think I love that you mentioned in the beginning emotional intelligence I think this is what it is about right so if you want to have humans plugged into the system we have to level to their multi-level I guess system values uh um I guess
2:08:51 · representation and that that is the difficulty that we want to to incorporate in this no I think I think it’s a it’s a big question I think uh Erica would would love your your perspective on what Olaf posed and and then if you can also further build on that um the distinction between like adaptation that serves human agency
2:09:15 · versus adaptation that optimizes uh engagement or compliance because because that can happen and I’ll give an example like with reinforcement learning with human feedback we have done studies where we’ve seen that uh people like we bring humans in the loop to to look at the responses from AI and then basically thumbs up and down like whether the answer is right or wrong because that feeds back into the it reinforces back into the learning.
2:09:39 · But uh we conducted some studies um and then we noticed that people were getting into the compliance where they were basically automatically agreeing with everything AI was basically uh uh presenting to them which defeats the entire purpose of human human in the human feedback in this case. So that’s an example of a compliance, right? So so we somehow about our our human nature it it pushes into that that compliance uh too quickly.
2:10:07 · So I’m just curious from your perspective and adaptation that serves human agency versus versus optimization for for uh engagement and compliance.
2:10:17 · Yeah.
2:10:17 · Um I love that we are wrestling with big questions here and no easy answers. Um I think you know this kind of takes me back maybe 15 years as different algorithms getting introduced and folks just saying well the machine knows best you know so whatever the algorithm is it is right. Um and this call for algor algorithmic transparency of when is an algorithm getting used.
2:10:48 · Yeah.
2:10:49 · To to make decisions that affect my life and um yeah the the tendency as Sher Turkl says like to expect more from technology and less from each other. So how do we undo this the machine must be so much smarter than me so I will always comply interaction?
2:11:10 · It’s it’s like we have a deep groove um in that interaction and and kind of as a sidebar um as I talk to folks about how are they trying to have more agency in their LLM interactions on an individual level. What folks are talking about is at the very least setting, you know, customizing the instructions of like always disagree with me.
2:11:38 · Yeah.
2:11:38 · um which not everyone’s going to ask for that. So I think it takes us back to a design like how can we design these systems in a way that invites debate or disagreement and isn’t just a constant yes and between user and LLM. Um because otherwise it’s right. We’re just like going into rabbit holes essentially.
2:12:08 · Mhm.
2:12:09 · Um and and deeply inside that like spiral.
2:12:13 · Yeah.
2:12:13 · It it it and then we are seeing the examples of that with examples of like psychosis and then things like that that are happening already. Um so so Erica like building further from that I know you mentioned that uh your initiative it’s around around measurements and bringing uh more humanity into the into the benchmarks and the measurements right and and one
2:12:36 · challenge I see in the current design thinking around benchmarks and measurement that they are also static uh they whereas uh you know intelligence we’re talking about continual and adaptive learning here and there’s an evolutionary aspect of it And and today’s benchmarks are not designed to to understand the evolutionary aspect of learning and and we talked about you know early childhood development from a monticery which is very play-based and in collective and collaborative learning um to we are talking about now uh
2:13:07 · reinforcement learning you know uh uh people who working in that you know giving their agencies essentially to the AI and just just complying with it. So somehow we shift from this playbook based learning over time to this this sort of a compliance-based approach. Uh I’m I’m curious to hear uh um uh the type of questions you will pose to designers uh of of these new benchmarks uh around evolutionary aspects of the benchmarks.
2:13:34 · Should benchmarks be uh be accounting for for evolution of learning and essentially of human learning over time and then perhaps also measure uh uh deviations into into uh compliance and loss of agencies and things like that.
2:13:50 · Yeah.
2:13:51 · Yeah.
2:13:51 · I think an absolute yes about you know measuring the compliance versus fullon agency. Um and and you’re absolutely right. So benchmarks are static and and I think we’re always playing catch-up. Um so it’s it’s easy to be about a year or two behind um the
2:14:13 · state-of-the-art and I think that’s where ongoing production monitoring again this is you know post- deployment but I think that is one part of of the answer here. Um and and I think the other is you know moving beyond benchmarks into um RCTs uh where we’re really understanding you know how to introduce the right amount of friction so that learning can actually happen.
2:14:46 · Um, so I know OpenAI has been doing work around that in Estonia and um, really measuring what that friction looks like. So I think it’s, you know, we’re very much we’re far away from, you know, one-sizefits-all, but I think that is one approach. And I’m I’m not remembering where I heard this. It could have been from Sal Khan of Khan Academy but um but could have been elsewhere that some results are showing that uh learning outcomes are enhanced.
2:15:18 · One is deeply personalized. Um for instance, if you’re learning from a deep fake of someone that you know and and care for or just in an area that you’re already passionate about. So yeah again multiaceted approach.
2:15:40 · No absolutely and I think I think it’s it’s it’s uh thank you for posing that and I think it needs to be multiaceted approach uh and dynamism into the benchmarking understanding needs to be brought in uh versus the static measurements that are out there. So Ming uh from your perspective as you are leading research and development around continual learning and adaptive learning systems and open-endedness within that. Um what type of benchmarks are you uh are you looking at?
2:16:08 · What gaps exist in benchmarks so that we we actually can more accurately measure uh this this uh adaptiveness of the learning from a childlike learning to to as it kind of evolves through various iterations. And then within that if you can maybe uh speak to uh uh agency versus compliance and maybe that’s a separate measurement when the one is more on the on the on the on the technical side algorithmic side and the other one on the uh human flourishing side of the things. Yeah.
2:16:42 · Yeah.
2:16:42 · For sure. So I think a lot of the historical progress on continual learning has settled on like very um clean and um in a close environment where people design a sequence of tasks and look at the learning progress in each of the task and whether there’s forgetting. Our lab sort of uh takes a different approach where we look at more realistic environments.
2:17:07 · For example, on videos, we directly take the long form uh child uh headcam videos for for continual learning. So, this consists of hundreds of hours of these long form video from a single perspective. And then we want to see like whether networks can learn from scratch uh by consuming these streams of videos by one pass through the video. No like um resample or replay.
2:17:35 · And there is some memory uh in the network of course but uh the overall learning trajectory is one pass through the video and see if the network has learned general visual representations but also multimodal learning by associating different objects uh with their uh visual appearances.
2:17:58 · So that’s kind of uh one approach that we have been uh taking this effort for the past couple years. And then the other is more related to LLMs. As you know, LLMs has been trained by all kinds of data out there. So when you actually do benchmarking, you never know like whether they have actually seen this piece of data or they are just uh doing true generalization.
2:18:21 · And also to um towards um testing the continual learning capability where we were taking at is to take the news events uh that happens every day and um you know form those as evaluation questions. So presumably labs that are training in the past hasn’t seen what happens today and that could form a way of testing how knowledge is continuously learned and forgotten over the timeline.
2:18:50 · So that’s also another type of effort that we’re trying and beyond just doing question answering we’re also looking at you know these forecasting markets how LLM agents are able to perform well in these uh settings um yep so I think that kind of answers um what we are have been trying on the benchmarking side of things uh towards u more agency I think what I’m
2:19:21 · um currently stands is that I think we probably need to understand better uh what agency truly means in artificial and biological minds. uh so do we uh what kind of the sampling or action process are we really taking or it feels like that we’re taking I think it they are kind of uh still very frontier
2:19:46 · questions in the research world and I think like by understanding that at least in artificial systems um can also um help understand ourselves and one aspect I’d like to bring up here is that a lot agency might come from just the diverse perspective, the historical trajectories that the agent has been through uh not necessarily at that specific time point making that choice of action. Maybe that’s in the form of a gausian distribution.
2:20:18 · Maybe it’s a softmax distribution. U but it’s not necessarily a deliberate choice at that very time step, but integration over the historical time steps. M Olaf, would you would you want to add to to to to that?
2:20:39 · Yeah, there there is a well there there’s a lot that resonates with uh with both. I think we’re converging on on on on some important features here, right? So, so the I think Erica is saying that the static benchmarks are are not enough and this so so we need to measure definitely uh uh different aspects of friction, dependency, compliance, uh agency agency very
2:21:06 · important here. um uh not only before but after deployment adapting as we go and and and Megan is mentioning a lot of uh concrete examples here uh showing that the continual learning itself uh needs to be tested and sort of
2:21:26 · more I guess in a artificial life perspective more naturalistic kind of stream and and and that’s close to how like you mentioned Ammer about uh children learn from embodied experience, right? Like not not from from just uh just sort of task sequences that are disembodied from the real world.
2:21:45 · Um and uh I I the way we encode uh knowledge can be can be very very tricky and multi-level and uh even in a group conversation like this actually I have trouble remembering who said what by by now. All right. and and and this is the the collective sort of mixture of of um of emergent agency that that we we study in the field as well.
2:22:12 · Um this is uh this is also part of the this is also part of the challenges I guess of studying those dynamics of learning as a whole system. uh when when this memory becomes distributed um you you also come
2:22:28 · it it comes with challenges and and and and and plus values but uh that means that the benchmark uh that you’re using uh can’t be a fixed you know static benchmark for that either um so so yeah I think this is a this is a strong convergence between between those uh those threads yep no there’s definitely and I think I I I’ll I want to switch a little bit to a um we we we talked a lot about uh
2:22:55 · learning and agency and uh uh to to flourishing and I think that the big question uh in my mind right now is that you know we talked we’re talking about plurality and and uh collective and uh uh provenence and agencies right so those are all at individual levels and um the question then is how whether human flourishing essentially can be operationalized uh without flattening uh the plurality of human values.
2:23:28 · Uh because we’re talking about you know building systems of intelligence that enable and augment human flourishing and there’s an aspect of that which is flattening and and and uh uh and in compression. Right? So, so the part the first question is is whether human flourishing can be operationalized without flattening and the second part of the question is uh if yes uh um u uh uh what are the what are
2:23:54 · the what are the ways or at least open uh questions of inquiry that should be further uh uh explored as research uh to to to dig deeper into. So maybe maybe uh I’ll ask with uh start with M and then to to to uh Eric and then Denafia.
2:24:16 · Yeah, I think uh yeah these are very deep questions. So maybe I will take a minute to think about them and pass over to other panelist first.
2:24:26 · Okay, sure. Who wants to take it first?
2:24:28 · Um Olaf or Erica?
2:24:31 · Go ahead.
2:24:33 · Sure.
2:24:33 · Um yeah, I think I it’s absolutely worth finding out whether they can be operationalized versus not. You know, I I think again this kind of stepping back and then stepping back in. We’re like living inside the hangover from social media that’s like a ongoing experiment that’s still with us. And and I I think many of us don’t want to see all of that repeated in our you know AI infused powered lives.
2:25:06 · Um hence this focus on flourishing which like you’re saying Ammer is pluristic. So there cannot be one definition. It’s yeah it’s many.
2:25:19 · Um and I I think yeah does does it equal flattening? I think remains to be seen, but I I don’t see much of it being operationalized yet. So, I wouldn’t say that we have a lot of data on it’s being it’s we we have effective approaches here or not, which I think gets back to some of those system dynamics of, you know, business model and and what’s driving things. I do just want to um have as a sidebar here.
2:25:50 · We’re talking, you know, like all day. It’s about agency and and there’s a lot of expertise on this panel and on all all the speakers um for this and there’s lots of wisdom and expertise outside of these you know walls as well and so I
2:26:12 · think there is a convening of the public the global public that needs to happen of you know how do they see what does flourishing look like for them, how do they see um their rights visav AI? So, I’ve kicked off an AI bill of rights um process where we can all name what are those things that that we think we really require and demand.
2:26:43 · No, thank you. Thank you, Erica. And we’ll definitely take I mean very aware of the AI bit of right. So, definitely take a look at that for sure. Uh then thank you for I know you Erica you have hard stops so I just want to make it uh the next uh I think uh Olaf and Mingi and I think if when you answer to this question if you can also add in how do we ensure that the the the uh uh collective systems of
2:27:06 · their open-endedness there is care built into it right so that they’re also uh uh safety conscious so so maybe uh Mia if you’re ready uh or or Olaf here yeah so I can I can share uh One of my perspective here is that um I think there are a lot of debates on what means to for the future human society to look like what means good what means flourishing and so on.
2:27:32 · I think one feature I think it’s worth thinking about is still the epistemic uncertainty. We don’t know what’s really good for us. So therefore we have to preserve the diverse uh intelligence out there just in case maybe there are something that we could miss that there’s no way that we can go back. So that’s what I wanted to share on this topic.
2:27:59 · Yeah, thank you. That that’s a uh that’s a big gap for sure to to to delve deeper into. Uh Olaf, uh last words.
2:28:08 · Yeah, so so it’s been very playful. uh I I I really like the this session. Thank you. Thank you guys for for for uh for for all those points that are are very important. So I think Erica your point now uh on operationalizing uh flourishing uh from the top down I guess I guess bottom up as well uh like we we said before uh there there are risks um that that we see. So, so I think that the the thing that I I fear the most is this flattening problem uh as well, right?
2:28:40 · So, u this is something that we should pay attention to and and Megan’s point also matters here. Um I think um yeah we we we need to model carefully the system and and and the representation will matter for for how we are going to um yeah h how we are going to design those actions for uh the the this collective agency um and and
2:29:05 · yeah the path being taken uh by uh by the system as a whole um is is affecting yes it’s also this will look very different for various applications Um but from my perspective I I I think uh there is there is something that we can do uh on uh not not metrics but uh I
2:29:28 · guess a sort of translational metrics between uh between perspectives and incorporating this diversity is the the thing that I would like to to pay the most uh attention to. Um so uh yeah this is uh yeah this is something that that we we can take into account for not not metrics but I think metametrics is the message.
2:29:50 · Yeah.
2:29:50 · Yeah. Thank you Olaf. Thank you uh Mi and Erica for joining us as well. Uh we appreciate all of your perspectives and uh hopefully as we continue this conversation you’ll also participate in in continuous uh research and and in papers as well. I think one learning for me for this session is uh two learnings.
2:30:10 · one is is uh play is important. We we somehow have forgotten the role of play in learning. Um I think that is so Erica thank you for bringing that in and then Ming I think what you brought in from an epistemism perspective is extremely crucial here. uh and and then and then Olaf, you know, uh bringing your perspective al together uh into uh this
2:30:32 · this meta approaches to understanding and measurements so that we build systems that do not just uh flatten but at least keep the sense of uh curiosity and plurality uh into the system because that’s where the care in the output will come from and hence the the aspect of the humanity. So, so, so thank you everyone and uh uh we will pass it off to Trisha actually.
2:30:56 · Yeah, we actually Yeah.
2:30:57 · Yeah.
2:30:57 · No, there’s a question from the YouTube um that Paul has surfaced which is uh what are the thoughts on how open-ended approaches square with safety or alignment or care? How do we know an open-ended system is safe and what does care mean for systems not based on rulebased ethics and judgment or alignment?
2:31:18 · That was a big These are like This is not just one question. This is like three three questions in one. I’m going to paste it in the chat so the panelists can see and get a fair opportunity to understand the actual question.
2:31:36 · Thank you to the person who who asked it. Who wants to take it on before we go to lunch? I think maybe that’s a question for Ammer because I I so so this way we can uh we can move on on schedule
2:31:52 · I think I think no that’s a very big big big question I think a very critical question so my personal thought again not not much backed by research again taking purely an open-ended aspect of it is that uh let me read the question first one time so thoughts on how open-ended approaches square with safety and alignment in care. Um so I’ll take that one first. And I think I think that today’s paradigm to how uh Thank you Ming. I appreciate it.
2:32:18 · Um today’s approach to to alignment and how it’s is is defined and structured right now is a basically a post-training behavior adjustments where philosophical perspective of a small select group of people essentially imposed on the models right at the post talk and and and again
2:32:37 · I may be generalizing it here but but uh with my limited understanding of the alignment here and what it does is that in my opinion is that it forces a perspective of a very select and small group of people and to the rest of the society and and as we’re talking about pluralistic of the society.
2:32:53 · So in a way what Erica was mentioning towards the end uh how can we talk about human flourishing without the rest of the humans in the room like have we defined it what flourishing is for one versus the other right so so the plurality for me and again within open-ended aspect of it has to come from that angle that alignment has to be defined uh at the root uh and it has to be uh uh uh
2:33:20 · localized it has to have its own localized perspectives and governance and participation and it cannot be in a situation where setup where uh one perspective of alignment wins over the other perspective of alignment. I think it’s it’s the the systems have to be defined in a collective uh collective ecosystem where perspectives are understood um by uh AI agents
2:33:45 · perspective can be shaped and shared and then pers fact uh and perhaps uh uh uh uh uh some sort of a perspective evolution that would come out of it that factors those in. So I think I think it’s taking more of a uh a pro-social approach to to designing systems uh within open indust ecosystem will will enable at least more safety and alignment and then uh and and perhaps
2:34:10 · essentially care in this case because a lot of the alignment would have to be would would bring in in in uh perspective things like our stewardship uh based communities or or indigenous communities versus uh uh uh uh and and and various other parts of the world, right? So, so in my opinion, I think it’s it’s the those various aspects kind of feeding into an ecosystem u and then collaborative colle collaborating and and uh and then evolving.
2:34:36 · So, so anyway anyway uh jumbled up thoughts but but uh that’s that’s my high level perspective on how uh it could potentially be done or at least it should at least be attempted to be done.
2:34:52 · Thanks, Amber. Well, c certainly we all agree we have to rethink alignment completely. And maybe that’s not even the word or the concept, right? If you’re actually getting to the source um and fixing the actual infrastructure versus like exposacto, which is how alignment is done right now. So, I think we’re talking about something completely even different structurally. Well, thank you so much for that exciting panel to follow uh the one that Olaf started us off with with Blaze and Pier. We’re now going to move all that play into some in-person time and we’re going to acknowledge that we have bodies.
2:35:23 · We’re going to take a break. We’re going to go take a walk. We’re going to send some emails and look at the sun and get a snack. And we will come back in an hour and 15 minutes after you have rested your mind and refreshed yourself. and we will go straight into session 2A which is the kind of the second part of the programming that you definitely want to stay for which we start to get into the actual um you know techniques of how are we going to get this done into some of the nitty-gritty stuff.
2:35:54 · So with that I will take us into a break. All right, welcome back from the break. Uh, I hope that we all took care of our bodies for those who have been here with us since 10:00 a.m. It’s now the afternoon on the east coast of 3 p.m.
2:36:21 · where I’m at in Brooklyn. And so this is the super uh this is the conference on super intelligence for humanity. I’m Trisha Wong and I’m your MC. And for those who just joined, drop your questions in the YouTube link and we will let the moderator know and surface your question. Now this summit is being convened by Cognacy, a public benefit corporation whose charter advocates for humanity aligned super intelligence. And what the hosts have done for this conference is gather their favorite thinkers and more like friends in this space to figure out how to get humanity from A to B.
2:36:52 · And A is where we are right now, which is a dominant bet in AI that has been all about scale and bigger models and throwing more compute and more data. And this bet has clearly clearly produced real capability. It’s what enterprises are all hungry for to try to figure out what is their AI strategy. But there’s a gaping layer that is just missing.
2:37:14 · And what we’re doing here is to talk about what that new B is, which is artificial collective intelligence or that’s kind of the general word we’re calling this whole field, which is to build AI that has what Ush said is dignity and sovereignty into the actual architecture, not just as a patch to it. And it’s about recognizing that intelligence is distributed and coordinated. It’s not monolithic and it never was, as Bla1 had reminded us.
2:37:41 · And it’s that systems represent the people whose knowledge actually makes the world work with context and authorship intact and governance that uh gives communities and institutions meaningful authority over their data, their knowledge and their cognition. And so the first conversation with bla1 and beer reframed uh with pure reframed this scale itself. The argument was that you know intelligence we admire in humans was never about the individual. It’s a collective output of society. And that was a really fun conversation.
2:38:12 · If you were there, it went over because it was just so exciting. We were talking about, you know, how intelligence actually comes from your interaction with the environment. And if you take the environment out, then it’s a whole different type of um, you know, cognition that is being developed. And then the second conversation we moved on to uh with Ammer as our moderator with Erica and Olaf and Men. And then they took us to the next step.
2:38:36 · They said, “Well, what would happen if since we know that, you know, we’ve established in the first panel that intelligence is collective and curious, well, what does it take to keep it humane?” And so, we heard several answers. Let me just give a quick recap because that we can then draw upon that for our conversation since today is all about building up to, you know, really establishing what are the edges and the bounds of this new field. So Mangay talked about, you know, preserving a preserving agency at the edges.
2:39:07 · Erica talked a lot about emotional intelligence and that we’re just nowhere close to replicating the way humans actually learn from uh in an embodied physical continuous way. And it was really cool because we got to talk a lot about play. Um Olaf talked about bacterias and how that interacted with, you know, playful learning and that when you’re playing, you’re not even um consciously learning, right? you’re you’re just learning through the process. But also, we talked about the dark side, which is that it’s very easy for AI systems to be extractive.
2:39:36 · And that’s why Erica talked a lot about the importance of measurement for human flourishing. But then Olaf talked about, hey, we have to be careful we don’t flatten that. And lastly, we talked a lot about emotional intelligence. That was the first time that concept was brought up in our panels. and that um how do you figure out what the constraints are of embodiment when you’re thinking about emotional intelligence? So now that leads us to this panel where the focus is on the languages of artificial intelligences.
2:40:09 · And so we have a great moderator here. We have Lipica Kapor from MIT who is going to be our moderator and works at the intersection of human- centered AI and decision making. And I’m going to turn it over to you Lipica and I’m also going to say I I am also on the panel.
2:40:26 · So I’m going to let you take it over and then I will let go of my MC role and join you as a panelist.
2:40:33 · Great. Thank you so much Trisha. Happy to have the tables turn with the session for you. So uh this session uh which is the languages of artificial intelligence and I want to uh spend just a minute on why this session is placed where it is and I think Trisha touched upon this and I just want to make sure that you know um you kind of understand what we hope to leave from the session.
2:40:54 · So within the summit’s broader frame of ACI this session sits at what I want to call the connective tissue layer and language is what makes shared cognition possible at all. So uh without shared semantic ground um agents and humans cannot coordinate or audit or hold each other accountable. So your provenence, consent, governance, institutional memory, all of it has to pass through language at some point.
2:41:22 · Which means if language is broken or ungoverned, every other layer is too. So here is the tension that I want us to surface uh through this panel is that on one hand language is the most successful collective intelligence infrastructure humans have ever built and roughly I think more than 7,000 mutually unintelligible systems evolved over millennia that somehow let strangers coordinate and on the other hand the
2:41:51 · version of language being built into AI systems right now is converging fast onto a monoculture mostly English mostly X and mostly trained on some corpora and uh increasingly opaque even to the people who built it. So the question for the next hour is um the so the question
2:42:11 · for the next hour uh not is not that is language important we know it is the question is what actually breaks when language becomes operational infrastructure between humans agents and institutions and uh what are the failure modes what protocols need to exist and what does accountability look like when meaning moves across boundaries it was never designed to Ross. Uh so before we get into that, let me introduce our wonderful panel here. Uh we have Bruce uh Schneijider.
2:42:42 · He needs almost no introduction. He’s a security technologist called a security guru by the economist, fellow at the B Buckman client center, lecturer uh at Harvard Kennedy School, uh author of many books including a hacker’s mind, one of the most influential public voices on security of complex systems for the last three decades. Then we have Damian Blassi. He is an ika research professor at the center of brain and cognition in Barcelona.
2:43:09 · A research associate at the culture cognition and co-evolution lab at Harvard Schmidt sciences polymath and branco fellow. He also advises UNESCO uh on linguistic diversity. His work asks what gets lost when we treat languages as if it had a default.
2:43:25 · And uh then we have Michael Springer. uh he’s the president of Sony AI and CEO of Sony research, a roboticist by training with more than 70 papers on language evolution, grounded communication and the foundations of artificial intelligence. And he’s been thinking about how machines learn to talk to each other longer than most people in this field. And last but not the least, we have Trisha. You already know. Uh she’s the co-founder and CEO of the Advanced AI society, building infrastructure for verifiable AI.
2:43:56 · She’s had a long career at the intersection of ethnography, human- centered AI and the question of what counts as knowledge. So I think starting with the first question for Bruce. So Bruce um if language becomes the infrastructure for AI agents and institutions, what are the main attack surfaces? Is it spoofed provenence, semantic drift, impersonation, hallucinated authority or something else?
2:44:28 · So, human language is sort of by definition ambiguous and I think the ambiguity is uh is certainly an an attack vector. Uh there was a a philos philosopher at MIT some years ago. I’m blanking on her name.
2:44:45 · It’ll come to me in in a couple of minutes. Uh Abby Abby Jacqu. what she said is that in human language and and even in thought, wants and desires are always underspecified. I mean, that’s the moral of the genie, right? The genie can always grant you a wish in a way you wish uh he hadn’t. That’s the moral of uh of a of of King Midas.
2:45:06 · And and I think that ambiguity gives a lot of room for AI both by mistake and because directed to do things you don’t want even though you tell them to do it. Uh that’s a new attack vector. I mean certainly there’s the other things you mentioned but I think that ambiguity which you don’t really have in a programming language in a formal specification language is a really interesting new way to think about uh semantic attacks
2:45:43 · and and oh I have a quick thing for well Bruce you know I think it’s ambiguous it’s more ambiguous the farther you’re away from that culture it’s actually less ambiguous if you are culturally situated I mean I I not I think it’s more ambiguous if I say bring me some coffee, right? You’re not going to bring me a pound of raw beans. You’re not going to buy a coffee plantation. You’re not going to rip a cup of coffee out of the person standing next to you and bring it to me. I mean, the kind of coffee I’m going to get depends on uh context.
2:46:12 · And if you if we’re in the same culture, you’re going to know what I mean by that. A Turkish coffee, an Italian coffee, American. Right. I think I agree with you. less is said.
2:46:26 · Yeah.
2:46:26 · No, I mean that’s my point is that you have to be culturally situated to to get that and some and you know some languages are even more ambiguous. So, it’s really going to be interesting to see how that’s exploited across different cultures, you know, for languages that cuz some languages are more explicit and some are not.
2:46:48 · And um so maybe let me bring bring you in Tam on this one like so ACI assumes knowledge can move across contexts and what does linguistic diversity teach us about what is lost when meaning is abstracted or translated?
2:47:06 · Well, I I I might start with perhaps what I think is one of the the biggest uh forgotten aspects of linguistic diversity that have to do with the subtle differences in in grammatical structure across languages and different languages might emphasize different pieces of semantic information. They might force you to obligatorily convey certain types of information. I usually when I when I speak to English audiences, I usually bring up the English plural, right, for nouns. Like English forces you to decide whether something is one unit of it or multiple units of it. Right?
2:47:38 · So you’re forced to choose a form. That’s information that you cannot escape unless you come up with a very uh torturous construction to avoid it. But there’s different languages that force you to express different pieces of information like when in the past an event took place or the future.
2:47:54 · You’re the degree of likelihood the epistemic nature of the statement you’re making. uh whether the reference you’re talking about belongs to one of n categories of grammatical gender you could classify the world etc etc and I think that you know I think
2:48:11 · there is a lot of emphasis on the fact that there might be not lexical equival equivalence between languages that the single word might not be perfectly I mean we all love those examples but the case of grammar is much worse because it’s not restricted to just one specific word but it permeates the unbounded space of messages that you can transmit So most of the time people don’t seem to care much about this.
2:48:32 · I think in part because most of the translation problems that we have been dealing with involve either languages are fairly similar from a global linguistic diversity point of view or they just happen to share a number of commonalities almost by chance.
2:48:49 · Um let me say for instance that in crosscultural psychology for instance uh there is an overwhelming representation of the comparison between Americans and Chinese and you know it seems like very different societies and people have been speculating about why these peoples and the languages might be different but from the linguistic point of view they share a number of important regularities like for instance both English and and Mandarin they have very
2:49:15 · little morphology so there’s a lot of things they don’t express and you know I could bore you to death with a a long list of technical equivalences between these languages. So I think that our intuition about how important these differences in grammar are is uh is is not based on a wide view of linguistic diversity. And again we might be losing those subtle pieces of information that are obligatory uh pacific to every message that we have to encode different languages.
2:49:46 · That’s very interesting. And I think would you say like that um what would you actually point to where the loss is most uh visible is it in between the the grammatical rules that kind of varies across different languages. Mhm. So let’s go back to this notion of ambiguity, right? Um there’s some extreme cases people for instance in academia they like to bring up um one variety of Indonesian called real Indonesian. This language has essentially no obligatory marking.
2:50:15 · You can say something like chicken person eat and this could mean the person ate the chicken. the chicken will be eating multiple people or you know there’s some general event that involves eating person and chicken. It’s completely undetermined. Now just imagine for a moment that you cast a message that is contextually grounded that is very dependent on the non- linguistic signal and you just want to translate it.
2:50:41 · Obviously there is deficit there, right?
2:50:43 · So any translation will involve putting in information that was not part of the message. Not only because it was not expressed but because users they don’t need to specify it right like there might be a case that you can construct a linguistic message where you don’t have to decide whether you want to say that something is plural or singular it’s just not part of it but if you want to force any message in this language into English you need to take a decision and the question is how do you base that decision and so yeah I think that’s um
2:51:15 · yeah and and Michael maybe u I’ll bring you in At this point, like your work on grounded communication asks how agents connect symbols to experience. What does shared context mean when agents perceive the world differently?
2:51:31 · Yeah, it’s a great question and I’m going to kind of add something to the discussion that was already happening.
2:51:36 · So one what Bruce was saying um so the way we talk about this is language is an inferial system. kind of gives people pointers and then you still have to reconstruct the meaning or actually the pragmatics also from the context and and similarly um as Damian was explaining uh languages have different grammatical structures that force people to kind of pay attention or conceptualize the world in different ways.
2:52:01 · And I think then the third step of this is actually uh conceptual systems that are really tied to the language that people speak.
2:52:10 · And so I I used to work a lot on uh spatial languages. And so there are these island cultures um off the coast of Papa Nigina who mark everything with respect to some volcano that’s always visible. And that’s a very different system from the systems that we have in English which are more projective with front, back, left and right and much more centered on individuals.
2:52:28 · And so this I think points to the fact that language has a very deep or it’s we’re going to be careful not to trigger any cognitive scientists but essentially uh language and meaning um are somehow connected and they’re very deeply connected and they force people to think about the world in particular ways especially once the language is uh established.
2:52:49 · And as you can imagine and this happens to all of us in translation uh we lose a lot of this um and uh if we do speak different languages then sometimes we can express ourselves better in different parts of the languages that we know uh and that applies to to words but it also applies to grammatical structure and so I think
2:53:12 · that’s the human side and I think that’s a really important property I should say that I’m always very impressed even by current systems of how you can mix and match languages that these systems have been trained on. At least this doesn’t apply to all languages of the world. But I frequently may interject English and German in some cases Japanese talking to these language models and they very often seem to come up with the right solutions.
2:53:38 · But um it’s interesting to think about what we lose if we do not have language models that do um kind of represent that variety that we see that diversity that we see in human languages because I do think that’s in many ways a strength um because it kind of broadens the approach to the world that we have and of course ultimately everything has to be related to context. So if you think about Japanese, Japanese often doesn’t mark the subject of a sentence.
2:54:07 · Doesn’t really tell you who is doing what because the assumption is that you can infer that either from a cultural context or the context that you’re actually experienc together with an interlocator at that moment.
2:54:18 · And what is important especially in the work that we did is that the context that embodied interaction that situation that in our case robots were in to to learn to communicate it helps um structure the the understanding of a meaning of a particular thing in a particular context. But it also helps in learning uh because if you’re not exposed to that context you actually can’t make those connections between the linguistic symbols and the things that you’re seeing in the world.
2:54:46 · So while the shared context is kind of like you know helping with the learning like how would you know that an agent uh or like yeah an agent had generally like shared context versus just fake the surface of it. How does that distinction do you think um agents tend to make?
2:55:04 · I mean ultimately you only we only had I mean we have only one measure in these experiments which is communicative success. So uh you know uh the interlocator is a speaker has some intention and then has some intention with respect to the speaker. In our case it was relatively simple referential type task.
2:55:22 · So like pointing to something or paying attention to something in the real world and then the speaker would test whether that actually has been successfully transmitted by checking whether that’s true and whether that is what um he he well in this case robots what they had in mind so to speak. um loosely speaking so sorry again don’t want to trigger any cognitive scientists here um and I think in many ways our interaction with AI systems are the same so if we work with AI systems to uh give
2:55:53 · us answers to things write documents for us do reports I think the ultimate test is does the system actually come back with something that’s meaningful or useful or interesting to me um and that’s really what language is about right I mean at least in this task oriented view Maybe not everything doesn’t cover everything that language is about. Language is also about inspiration and so on. Um but if if that loop is closed and people are happy with the results, I think that’s a very strong signal that things are working.
2:56:24 · That’s a very usage based view by the way again. So they’re they’re very different cognitive views on this. So cognitive science views on this.
2:56:33 · Right. And Trisha onto you next like from your work on quantification bias in human meaning like where do AI systems most often flatten or distort the tacid knowledge?
2:56:47 · Yeah.
2:56:47 · So the quantification bias is a concept that I had coined about um probably like I don’t know 15 years ago when I was at the working at a research lab at Nokia and um I have a whole talk about this online that I did essentially 10 years later when I felt was safe enough but I had was working inside Nokia and I saw the quantification bias which is the um the tendency to prioritize that had that which is quantifiable in the form of numbers over anything that is not quant.
2:57:16 · quantifiable. And so this is a problem that started way before AI. Um I think some people talk about like these systems are imperfect now, but they have been imperfect ever since we’ve started digitizing data. And in particular, what I saw happening in enterprises is that there was this idea that we can now have big data and we can now make all the predictions and decisions, all the correct decisions we needed. But the problem was that I saw up close what happened when a company uh would only prioritize quantified data.
2:57:45 · And I had uh been living in China um on and off for about 10 years then. And I had been doing research explaining to them that you are you know it was in 2006. So it was like Nokia was the top cell phone company. It was Apple had just come out and everyone thought this is ridiculous when you saw Microsoft making a joke out of you know Apple. It was like Steve Palmer remember like that video of him jumping up and down on the stage with like sweaty you know shirt and everything.
2:58:10 · And I remember coming back after um about a year of research and I was like, “Hey, Nokia, I need access to more quantitative data because I have a lot of thick data, the qualitative part.” I had to rebrand qualitative to make it just as sexy as big data. So I was like, I have this thick data um and
2:58:29 · it’s telling you telling me that I don’t think Nokia has a business because I don’t think anyone’s going to want your phone anymore. and explained to them the whole emotional um experience of a phone and they didn’t really want to listen and so anyways I left Nokia and then eventually I did a whole TED talk on that but what I had learned from that experience has cascaded across all my work with any kind of emerging tech and especially with AI so it’s like my work hasn’t necessarily changed but it you only what you see with AI is that it uh
2:58:59 · it only increases and scales the flattening and distortion of tacet knowledge because tacet that knowledge is very resistant to quantification and but there but there are people who really think that you can perfectly quantify everything and put numbers and weights to everything but there are certain types of fields of knowledge that are more easily quantifiable like code you know there’s um that is much more quantifiable than a lot of the pe construction workers I was living with or the migrant workers I was uh spending
2:59:30 · time with in Mexico you know so I’ve I’ve done this work all around the world to find about you know um what is the kind of work that is very difficult to quantify and one um the system rewards what can be measured that’s just you know and it ignores what it can’t that’s what you see happening in AI systems at least right now right with the current systems and um I and and second you
2:59:53 · really see that the system flattens the people who have given it these um inputs which are oftentimes which are anonymized and what what I see happening is that when executives are working with AI is that they really think it’s real.
3:00:08 · Especially with LM, I want to be very specific about the type of AI we’re talking about is that um they that there’s this illusion that the response is because it’s happening. It’s it’s it’s in a language coming back. I think that’s the that’s the issue is that we’re giving language inputs in and it’s flattening it. But then what’s weird is that it’s the system instead of a dashboard like before with um big data, now we’re in AI. the system is now giving back language.
3:00:34 · And so it’s really misleading people to think that these systems represent the exact reality when really it’s it’s not. It’s not representing facts. It’s representing at least on NLMs, it’s representing probability. It’s a probabilistic system, but I see executives um and and people using it as a deterministic system.
3:00:56 · And then another way I’m seeing it is that the system um I think it just has there’s no mechanism for people to take to correct um or revoke um and change the language of AI. So there’s no there’s no kind of expos once that data has been gathered and ingested um there’s little ways for people to interact with the system.
3:01:18 · And so in the end um it it just means that the system is reflecting um information that may be outdated. it may not um people the people who are being represented may not agree with it and so I see like a real distancing and the quantification bias is just even more extreme and it’s if anything it’s hidden now it’s it’s hidden through language
3:01:44 · that’s a really interesting point and I feel like uh like as humans we separate the buckets of what is deterministic and what’s probabilistic deterministic because you know there are rules and we we know how you know what comes next and however I think for AI uh they are still differentiating they kind of treat deterministic and probabistic in the same way even though the rules are defined it’s kind of uh doesn’t take it you know at 100% probability that it is what it is supposed to be so um yeah
3:02:16 · thank you for the thought sorry was someone trying to come in all right so uh moving to the next uh you part of um this panel today. I would love for um each of you to maybe name one specific failure mode in language mediated ACI uh which is something concrete. Um maybe Damian we can start with you on like how do we perceive diversity while still enabling interoperability.
3:02:50 · Okay.
3:02:50 · So I would use this as an excuse to talk about a topic that I’m very obsessed with and uh there is a a wave of papers that started in 2024 with the famous do lamas think in English uh where essentially people have been looking at the intermediate layers of LLMs and just trying to keep track of the tokens that are being activated and mapped that into the language and one very robust finding of at this point I would say like 20 or 20 plus papers in this tradition uh all very well cited is that in the intermediate layers of of
3:03:23 · LMS there is a clear activation of the high resource language. So some aspect of the computation takes place in that language and then only in the last part of the networks is where the translation takes place. There’s also something that happens early on we’re still not very well known that doesn’t seem to project to this the space of tokens um which is
3:03:43 · very interesting to discuss but I think that’s that’s something that we are just starting to to understand right the fact that yeah we empower and we resource languages through the most high resource languages out there and and they seem to be working fine for a number of very pedestrian uh uh uh tasks but they’re
3:04:04 · build this huge mediating filtering that is a high resource language that it’s you know presumably uh involved in the the loss of nuance in in the loss of performance in certain tasks that we are barely talking about. Um, another thing I want to say in this regard is that um, and I think that Michael was sort of hinting at that is until recently there was a tendency of discussing linguistic diversity and AI from the point of view of deficit, right?
3:04:32 · The fact that we’re not serving very well the world’s languages. So this is a problem. Um, but I think there’s another aspect of this which is to think about this as a as a untapped resource. I know and it has been mentioned a number of times. Languages have been slowly evolved over thousands of human generations. They all imply slightly different representations of the world, different vantage points.
3:04:59 · And presumably we want to have more of that, right? We want to have as many diverse perspectives of the world as possible through languages. And presumably those could fuel an increase in machine intelligence across a number of tasks. And I know I’ve been a bit vague on what task I’m referring to, but perhaps we can zoom in later. But um you know, I will start with that.
3:05:19 · So maybe I can uh just follow up Damian here. Um because I think that’s exactly right. So I think one of the strengths of human cognition is a diversity and it’s I think it’s evidence by language.
3:05:29 · Um and of course one of the worries I have with maybe current AI systems is that uh we’re sort of overfitting onto a very large probability distribution of a current language, right? And and for a while at least the models were trained and then they stopped and then there was a lot of stuff that was um kind of left out for the training. I think that’s gotten a lot better.
3:05:49 · But I feel like one of the things that maybe this is not leaning into enough or maybe can make some progress on is that language also I mean we we do also create language on the spot within certain limits in order to for us to make sense of new situations and this sort of very strong adaptability right in in a background of like some form of uh cognition abilities that are shared across humanity. I think that’s a really interesting one.
3:06:17 · Um, and I’m also thinking that so yeah, there are sort of the interfaces between humans and AI systems. Um, but there also interfaces between AI systems and they don’t necessarily have to be the same. Um, and I think there’s there’s some strength there in in really forcing much more adaptive systems that can create new conceptualizations and new new ways of making sense of the world on the spot and then test them out. So, that’s something I’m really excited about.
3:06:47 · Can I can I respond to that? I completely agree and I think this is something that I I fear sometimes might be misinterpret of the not in this case but in general when I say you know different languages or different pathways to cognition etc. There might be this idea that there are like hard boundaries but what Michael is mentioning is extremely important which is there is also this huge meaning making generative capacity that humans have. Let me just to link back to what Trisha was mentioning which is enography uh which is a field that that I love deeply.
3:07:16 · There’s this project that we are currently coding data for that has to do with first encounters between humans uh who don’t share a common language. It’s mostly like uh um you know missionaires and concistadores and uh people who got lost in different parts of the world and you know human groups get together they don’t share linguistic codes languages that might have diverged maybe 40,000 years ago 50,000 years ago whatever and they’re still able to communicate with
3:07:44 · sometimes brute communicative success extremely complex messages like where is the river where this ship went last week and there were three people and you can still communicate even in the absence of language which I think is very important and that that’s again something to to highlight which is yes we have all these languages but apart from that fueling that creativity and ingenuity there is this open-endedness nature of humans that are able to establish meaning even without a common cultural protocol
3:08:15 · and and you know you really see this Damian I love I love first of all thank you for knowing what ethnographers do because I’m always like yay to the field we’re so such a rare breed and tech. But to your point, this is why I think studying memes is so important. You know, Kenyatta cheese made know your meme which is now in the Library of Congress. But he really formalized the study because memes are the smallest unit of collective intelligence that survives transmission. And you can also see how it’s generative. You know, memes were doing this before AI was here.
3:08:42 · And lang you see how language is the infrastructure. memes are the packets and then they they move you know through our pipes and they’re able to encode context without flattening. Now it doesn’t mean that you know it’s always the same meaning or that people can agree on everything but there is a collective um agreement and intelligence
3:09:04 · around memes and oftentimes I think we don’t take it seriously because it’s like memes about you know pop stars or from movies but you really see how a meme can carry emotional charge you know there’s countless BTS memes you see you know BTS pop stars like they are able to transcend language and culture Mexico City just had one of the largest gatherings for BTS and Claudia Shine bomb is going to you know invite them back as you know these Korean pop stars.
3:09:29 · So you see how those you know memes have played such a big role in creating a a collective kind of norming around that you know we may not speak the same language but we’re going to generate similar meaning and so it is possible and you see that it is tacet knowledge in motion with memes and that’s something that I think AI systems have not come up to. I’m not saying it can’t, you know, which is why we’re here to talk about that. But, uh, I love your point about that that yes, human human beings have that capacity and we can see that in the way mimemetic culture works.
3:10:05 · Uh, I I want to not stop Bruce from from jumping into this, but um maybe maybe I I’ll there’s another thing that I thought while Trisha and Damian were talking. So, and I’ll give you an example. So, we built a robot that is uh competing with um the best table tennis players in the world. And we did something similar for racing.
3:10:23 · And one thing we learned from this when you have systems that are approaching these really top performances of elite athletes is that if you’re an elite athlete in in some sport, there are probably like 10 to 50 people that are in the world that are at your level. And that means that the space of top level tempo tennis is actually largely unexplored like the kind of skills that people can use the strategies and it’s the same in racing.
3:10:52 · I think it’s basically true for any endeavor where um humans really need to work very very hard to excel. And if I think about like one of the obvious biggest human endeavors which is science there’s really two things coming together. One is of course I just looked this up actually there about like 1,000 to 5,000 of the world’s leading biologists and of course biology as in
3:11:15 · any other science is largely based on building a representation system and language to make sense of the world and what that means I think is that most of those areas mathematics physics and so on I think are actually largely unexplored and biology is a great example of this because it’s so complex and so if we if we had systems that can participate in this representation building together with humans and help humans better understand and explore that space.
3:11:41 · I think there’s a huge opportunity here for AI to and these collective systems frankly speaking to to unlock like um all sorts of stuff that we might be interested in um like cures for diseases or new forms of thinking about physics or mathematics and so on. So I think there’s two parts to this, right? One is we want to be understood and we want to create meaning in the world, but in some cases also we’re looking to explore the world with language and I think there representation plays a huge role and if systems can help us do that collectively that would be amazing.
3:12:17 · Bruce, would you want to come in on the on the failure mode here?
3:12:22 · So I I I have I think of a different failure mode which is really a little bit what I talked about earlier and that’s the uh what what I’ve heard called reward hacking. This is not necessarily a large language model. These are this is just other AIs where they solve a problem in a way the designers didn’t anticipate or intend.
3:12:42 · Uh there are a lot of really fun examples. You ask something that that’s fun. Reward hacking is great fun. Uh, I there was a a block stacking. All these happen in simulated environments. And so the block stacking simulation where the AI figured if it flipped the block over, it would get the points for stacking without actually stacking. Uh, there was a uh there was a boat driving simulation where the AI spun the boat around really fast instead of doing what it had to do.
3:13:11 · My favorite was some kind of evolutionary simulation where the goal was to uh get to a distant finish line as fast as possible and instead of growing longer legs or bigger muscles, the AI got really tall and then fell over. And these are sort of great examples of the AI thinking out of the box because it has no conception of what the box is.
3:13:35 · And that is either creative or it’s disastrous depending on on context, depending on how it’s being used. And so to me, those are those are fun when the AI misunderstands in ways that are interesting. All right. Um I think I’m also uh is is there any other comment that any of the speakers want to make? I have a few questions coming uh from the YouTube.
3:14:17 · You should take questions from the audience.
3:14:20 · All right. So uh the first one is if there are voices inside LLM the James Evens paper should we talk to the LLM or to these voices? Is there a right level to be talking to? Do we know the difference?
3:14:45 · It’s a It’s a deep question. I don’t think anybody’s like talk to everybody.
3:14:48 · I don’t know.
3:14:54 · Yeah. Especially Especially like in the early days. Yeah.
3:14:58 · Um All right. We have another one.
3:15:07 · So the alternative conceptions uh of AI that were discussed this morning focused on AI that is local contextual and continually learning can fit with a more heterogenous linguistic ecology.
3:15:20 · How might those sorts of architectural shifts, the kinds of concerns related to surveillance and the undermining of democratic systems that Bruce uh has warned about?
3:15:34 · Uh, I’m going to start, but I actually want to hear from the other panelists as well. I I mean, there there’s there’s this this theory that embodying these systems will give them an extra piece of learning that that having them have, uh, skin in the game, uh, consequences.
3:15:50 · Um, uh, Media Lab person, a really good Atlantic article, uh, Deb Roy, really good article on this notion of consequences. and that an AI needs to feel consequence otherwise what does it mean when an AI apologizes.
3:16:06 · So there’s some really interesting research here in that it will make a difference but you know uh a lot of people will say that that’s just doesn’t make a difference at all. And the answer to your question is we don’t actually know. And right now we’re experimenting with everything. But it’s certainly a worthy experiment.
3:16:34 · Um I’m going to jump in here. Um yeah so I I think it even goes like there’s a different way to look at it is to say yeah the obvious answer is like of course we go hyper local and we go personal on device or anything you know that allows us um to do what we were talking about earlier today right this more local contextual continue learning um and it should you would think that it would reduce surveillance because you don’t require everyone’s data to flow into a centralized place.
3:17:04 · Um, but then it’s not the for the deeper answers that surveillance problem isn’t really solved by where the data lives. It’s solved really by the identity that the system requires you to present. And so a lot of um the work I was doing was really is when I the reason why I chose China was it just so happened I spoke Mandarin. So that was helpful.
3:17:26 · But a lot of my field work I chose to do there because I wanted to be in a place where privacy was not the norm but anonymity could be was you know and that those two things are not the same and privacy is when the system knows who you are but cannot use that you know information against you but and then anonymity is when the system doesn’t know who you are and both have a place so at a bar you can be anonymous um and you can interact you know and share information but not share your identity and you can have privacy with a government because the government, they know where you live.
3:17:58 · They you pay taxes, they have your address, but they don’t see what you’re doing inside your house. So, you know, this is it’s really interesting. I want to go after Bruce who’s talked a lot about who talking about democratic erosion. And I I think, you know, it’s going to be interesting to see what current AI systems or even new systems that we design, which is um will it require persistent traceable monetizable identity to function? Um is that a property of the model? Will that be property business models and the architectures wrapped around it?
3:18:27 · And so ACI architectures could shift this uh but only if they’re designed to support this kind of context appropriate identity and give people the flexibility um to to choose you know and I think one of my my critiques is always has been before AI happened is that we have been demonizing anonymity and our spaces to be anonymous on the internet um and even
3:18:53 · in physical life on the internet in particular have decreased and so it’s something that I’m I’m watching, you know, is to see like how how do we have in order for these um democratic systems to survive, do we we have to have things like anonymity? So, how do we bring that in? You know, I would love to hear from others too on how they’re thinking about this.
3:19:20 · I could I could say something in terms of the again my my boring and recurring system which is languages. that uh there is there has been an increased interest in um on the type of vulnerabilities that might come with uh tapping
3:19:36 · different languages and I’m thinking just about prompting and guard rails right and one of the most interesting observations and I’m not sure if this is uh this is published I’ve seen this in a in a workshop two months ago where people have been prompting LLMs in languages allow for honorifics and where the language itself allows for a larger social distance, right?
3:19:56 · Their languages are relatively egalitarian where you can ask quote unquote neutral neutral question and there’s going to be a cost for a human and a machine to just turn you down and say, you know, cannot answer that. That’s not that’s not possible. But in other languages, the cost of rejection of just saying no implies a dramatic uh uh uh loss face.
3:20:20 · Uh so it’s it’s it’s much harder to deny a request in a different language. And what might happen is that in those languages what ends up happening is there is some concession. They can they they don’t jump the guard race completely but they give you something.
3:20:35 · And the question is well can can we use those vulnerabilities to access information that in the languages for which we do our LHF for instance are heavily guarded but they might there might be back doors where you can access the information otherwise. I mean I I I’ve seen research where guardrails in one language don’t work in another where you could translate the exact same go I don’t know if that’s the what you’re talking about but that is a similar uh issue.
3:21:02 · Absolutely. Here I think the point is that people are using their understanding and science of language to find exactly those languages where rejections are particularly costly from a social point of view. And of course if you’re Yeah. So, um, you’re right. I want to see that paper.
3:21:19 · That’s really interesting.
3:21:23 · I’m not not sure I can I can add much to this. Maybe I can add a practical perspective on guardrails by themselves.
3:21:30 · So, like as obviously I’m part of a of a company very difficult for us to trust guard rails today. there’s so it feels like it’s such a nent early space and there’s not enough kind of conviction um to to really trust guardrails. I mean literally um I heard a story recently from from from some internal teams that they were able to uh take out some guardrails but by asking the model to ignore the guardrails. So you know those kind of things are still happening. I think this is very early space.
3:21:57 · I think the models are really complex and and how to make that actually work I think is going to require a lot more research.
3:22:04 · I think on the privacy side it is such a large societal discussion that’s happening also today already with um like different types of um targets that people are trying to pursue. So as I’m sure all of you are aware of um issues around miners we’re interacting with AI systems uh which may change view of um privacy or anonymity. Um, so it it feels like it’s such a such a complex emerging space where there’s a lot more societal discussion that that has to happen.
3:22:36 · But I I’m pretty sure that um thinking that a local model does not have any privacy issues is a fallacy.
3:22:47 · Yeah, totally agree. It’s it’s like that’s to it’s not about local. It’s AI where the person interacting with it gets to choose what kind of identity they’re presenting and that and I think that system has to honor it also cryptographically not just by promise which is moving into Bruce’s field and I would love to hear his thoughts on that later if we get into that
3:23:09 · and maybe to chime in here I think um I think Damian’s observation about honorics is striking and then it suggests that the same model could behave very differently depending on which language uage it is prompted in uh which also sounds like a security problem than just a linguistic one. Uh Bruce would that change how you think about red teaming? Uh right now most red teaming is done in English.
3:23:34 · So so that so this is a known issue. I mean we were already I already know groups that will use alternate languages in in attempts to get around guardrails.
3:23:44 · So this is not something where we’re uncovering here. This is something that is known. uh I mean it’s it’s a subtle problem because you need enormous amount of expertise now or rely on machine translation which has its own issues. So so this does this is known and yes it it is a problem but you know prompt injection is an unsolvable problem like we know that this is not something we can you know need to figure out all the ways to do it and block them. It is fundamentally unsolvable with current architecture.
3:24:15 · So knowing that kind of helps you when you go into it, right? I’m looking at other questions here. Um maybe um so maybe just to talk a little bit more about the verification layer like what is the minimum verification layer um institution needs before they can rely on AI mediated language?
3:24:49 · What would that minimum level would look like?
3:24:59 · I really want to hear Bruce’s answer.
3:25:01 · Yeah, I know. I think about this a lot.
3:25:04 · And the fact that I don’t have to say I think about this a lot. The fact that I don’t have an answer I think is very telling. So this is the whole notion of integrity, assurance, trust.
3:25:16 · uh you know we don’t know how to measure this how to think about it how to determine if a system is above or below a threshold or what even that means so we are very early in our conceptualization of and I think it’s a critical problem and I really worry that our use of these systems is going ahead of being able to answer that question but we actually can’t answer the question right now we don’t have we don’t have enough language to talk about it in a useful way.
3:25:51 · Um, well, I I want to jump in here in that um, you know, the reason why we created the Advanced AI Society was to acknowledge that there’s a whole growing field called verifiable AI. And most AI is seen as scary right now because you can’t confirm a system actually did what it said it was going to do and AI said it was going to do. And um, we need to start somewhere.
3:26:11 · And there’s this whole emerging field that, you know, Bruce has great expertise in, which is cryptography. and I come from also the blockchain space and there’s all these technologies that are out there that are now being applied to AI and it’s an emerging field and it’s called verifiable AI and I think that right now we don’t um there’s no way at least in common at least in mainstream enterprise AI that’s launched is that there’s a humongous verifiability gap um and as
3:26:41 · the as you know Christian Catalini published a paper um he’s based out of MIT and he his paper along um with two other co-authors talked about how as the cost for compute drops you know the verification remains you know is a bounded issue and there’s different ways you can verify so that as the cost of compute drops the verifiability gap grows and you can verify by humans with
3:27:07 · the human being you can and this is not just meaning human loop it’s like literally we need to have a human say I authorize this but we’re talking if we’re talking about agentic AI we’re having you uh transactions are being done at machine speed and a lot of the mechanisms that we have like OOTH was built for a human initiated world for resolving identity. We never figured out identity to begin with. So first we have to give ourselves all a break because like the systems we we’ve been patching identity together.
3:27:33 · Um but I I can’t imagine a more urgent time right now to say hey all the all the tech we have been developing in the cryptography space. And the sad thing is I think it’s unfortunate um is that cryptographers and also all the people who worked on the technology for decentralization decentral um uh blockchain and crypto rails have been very separate from uh mainstream you know developers that have created our current AI systems and we can’t afford to have those communities be separate anymore.
3:28:02 · Usually the CISO is not at at a board level and they’re not at the very they’re they’re just there seen they they’re often seen as like anti, you know, um like they’re seen as like anti- innovation and so a lot of times people don’t want to talk to the security people until they have to.
3:28:21 · We can’t afford to do that anymore.
3:28:23 · What what’ you say, Bruce?
3:28:24 · That’s the way it goes, right? We’re always we cannot. So this is why I’m like we have to you know this is why we started we co-founded um a few of us advanced EI society to ensure that we can’t have this gap anymore and so to answer your um to answer the question of like you know what are the things that have to be resolved is we have to resolve identity we have to resolve authorization everything has to somehow roll up we’re
3:28:49 · going to live in we are in a world where human um agents will outnumber human beings but we have to be able to say this was rolled up into human this is human authorized this is what human delegation looks like and earlier um we had a talk I think it was um someone had mentioned all the legal work that’s now being done on how do you think about legal personhood for agents that all of that has to roll up to humans um and we have to figure out the identity piece we have to figure out the authorization piece so one of the first things we’re working on as the society is a proof of
3:29:18 · control standard for cryptographic verification because there are things we can agree to and it just that’s a thing that hasn’t been decided what what counts as cryptographic verification and we don’t want to let any one vendor control that and and and Trish Trish’s point it’s a thing to thing authentication it’s not me to you know usually when I authenticate it’s either me to an object maybe I use my face or me to a remote service this is a thing to a thing right it is a sensor authenticating to a car
3:29:47 · it is a car authenticating to an AI that’s in the cloud analyzing traffic we don’t know how to do that at scale that’s actually extraordinarily Bruce, this is why I want to get you involved in our work because I’m so happy you’re on the same panel.
3:30:00 · You and I will talk.
3:30:02 · Yes.
3:30:03 · It’s so interesting because I think and and maybe this is I think we’re nearing the time also, but I would love to hear all of um your thoughts on I think as we kind of you know wait to kind of uh get more clarity on how to uh how identity and authority like the rules around that becomes clearer like where does what do you think like where does the meaning actually live in a multi- aent system?
3:30:25 · Is it and it’s kind of I think not an easy answer again but is it in the agents themselves or in the protocol between them or how do you pass you know the the question of authority or identity between the two the agents themselves versus the protocols between them.
3:30:46 · Damian I see you smiling. Maybe do you want to chime in on this?
3:30:50 · Well, u what I have to say is that this is this problem is structurally the same problem that people have had with the ontology of language. Is is language and meaning something that lives in people’s heads or is it the convention that allows for meaning to flow? And I have to say that uh with the amount of time that we have left, this is still an unresolved problem. Um but it’s an interesting one.
3:31:15 · Yeah, I think this is one of the deepest questions. Where does meaning live? So I don’t think we’re going to solve that on this panel. I think uh one way to make progress in this is actually what who does it matter to. So um you know that might determine a little bit where people might look for things. Um I was
3:31:32 · going to go back to this verifiability question because um I mean we do as and I’m not saying this is a good thing uh just as a description we do run very large societal systems on nonverified systems right and I think one large example of this is really finance um which where where there’s a lot you know large interconnected systems computerized trading and so on that are all not verified and I think as an outcome of this we’ll see we we are seeing periodic large crises emerge uh
3:32:02 · where we then try to put in all sorts of mechanisms kind of post hawk like circuit breakers or you know market shutdowns for periods of times and so on. Um, so I’m kind of curious if this is ultimately going to result in something similar where uh rather than being able to build things verifiable from the start, we’re going to try and sort of post hawk govern these systems.
3:32:30 · It’s very interesting T. Um and I just feel like I think um just kind of going back to the other question once again about the where does the meaning live and I I agree like it’s it’s not a question that we can we can solve uh at this uh point of time but um
3:32:50 · I feel like this is a question that we we want to continuously like continue to think more and I think more of the research work uh will help us uh you know get more clarity on that and I think part of The purpose of this panel is to also you know kind of raise these questions that we understand or we don’t understand and continue to kind of work through those. Uh we have I think last two minutes here. If anyone has any final comments before uh I wrap up.
3:33:29 · All right, take that as as a go for the sign off.
3:33:35 · So I would say I think this has been like a really interesting discussion and I think um I would say for the wider summit the three protocol requirements that I think uh we can kind of leave on the table is that I mean one every AI to AI you know message should carry provenence and consent metadata. So authority is traceable. So I think we can agree on that.
3:33:59 · And then second that translation across context should should attempt to log what is what was lost and not just what was conveyed. And I think that’s um the diversity of meaning is a feature and it’s it’s not a bug. And lastly, I would say that authority claims in AI mediated language uh needs a revocable trust and that the static trust does not you necessarily scale and these are not solutions. These are just the beginning um of an agenda that uh needs more work.
3:34:31 · So thank you so much uh Bruce, Tamian, Michael and Trisha for this amazing panel.
3:34:40 · Thank you.
3:34:41 · Thanks Lepa. Thank you.
3:34:43 · And I’m gonna take over my role as the MC. And I’m so excited for all my co-panelists. I’ll be contacting.
3:34:51 · You should get a hat and change your hat when you do this.
3:34:54 · I’ll change my shirt.
3:34:58 · Yeah.
3:34:58 · I need like a special face made for a Zoom face made for me. Um well, I I just, you know, this is one of the I was so excited for this panel when I saw everyone’s names on it. So, and Olaf was right because he’s like, “You’re going to love his panel we put you on and such a great moderator.” So, with that, welcome back from the break um or welcome back from welcome here for the first time um if you just joined us and if not then you got to witness a super exciting topic.
3:35:23 · Um a really fun panel that took us into the second part of the day with of the gathering which is really starting to figure out what are the things that we’re going to need to design into our system. And so for those who are just joining us, the summit is being convened by Cognacia Public Benefit Corporation whose charter advocates for humanity aligned super intelligence. And this is the reminder that this is a gathering of friends to figure out how to get us from A to B.
3:35:48 · A is where we are right now with AI, which is like big models AI, where you throw more compute at it. where in the future is where we’re going is artificial collective intelligence ACI that builds upon dignity and sovereignty uh of human beings but actually doing that into the architecture. And so this next panel is a humongous one and this is why we have Olaf moderating it because it is such a meaty topic. Now feeding into this panel were three very important discussions.
3:36:22 · So we’re after this we’re going to be about halfway we’re already halfway through the day but the early morning with blaze and pure really started out with us just all agreeing I I don’t think anyone disagreed that you know we um intelligence is never is not individualistic even though we may portray it that way but it is part it’s a collective output of society and it’s that curiosity that drives the production of intelligence and that makes intelligence generative.
3:36:46 · So we have to protect that and that’s why in panel two we talked a lot about play about how do you keep that um you know collective intelligence um how do we protect it for human beings and so we talked a lot about the importance of emotional intelligence in the creation of um these kind of systems and how we have to be careful to not create extractive systems now in the last panel if you were not here I’ll um just a quick summary is that
3:37:20 · and to by the default our current systems have captured most of um through LLMs is through textual language and what gets lost and what gets flattened and in particular what type of tacet knowledge is lost and then we were able to talk about the governance piece which is really critical for ACI world which is that if we want people if our assumption is that we can develop a new world where we can capture more intelligence in a way that’s going to respect human dignity. That means that one we know that most knowledge is not being captured by our current systems.
3:37:53 · But in order to get this whole world of other knowledge that exists, we have to think about not think about we have to build in the governance piece which is the verification piece. And so we’ve got it got into the nitty-gritty. We had Bruce Schneider on it who is this you know star cryptographer. And now we’re gonna get into well what does it mean for session to be to talk about diverse consciousness and multi-cale minds. And I was really excited when I also saw Divia Chandler’s name on it because I’ve been following Divia’s work for a while.
3:38:25 · Well, Olaf is going to be the moderator of this session. Um, Olaf is one of the co-hosts and we also have Yashua Bach who’s executive director of the California Institute for Machine Consciousness, which is where a lot of the people are at right now in San Francisco as we have that view. So, hello to all the San Francisco people.
3:38:45 · And then we also have Phipe uh Beodun from the founder of law zero who’s building safe by design AI and cognitive societies. And we have Riyota Kane who is the founder and CEO of Aria and lead Japan’s moonshot project on brain machine interfaces and he’s one of you know the world’s leading experts on artificial consciousness. So it’s really cool that Olaf that you are now bringing in all the folks from Japan.
3:39:11 · And lastly, we have Michael Levven um from the Van who’s a Van Ever Bush distinguished professor of biology at TUS and also director of the Allen Discovery Center. Exciting. Um I’m excited to see him and Divia exchange on the work around biology and his work on biomectric signaling as a form of when cells engage in collective intelligence. So with that, I’m going to turn it over to you Olaf.
3:39:40 · Thank you Trisha. So uh we are in good company here and hopefully people are uh also uh hearing us fine online. Yes.
3:39:51 · Yes. All good.
3:39:52 · Amazing. Amazing. All right. Uh so this is a sort of relaxed session on a very relaxed topic. The nature of diverse consciousnesses and multi-cale minds. Uh very easy questions. Uh so so of course we should relax uh in a niche construction relaxed way. Um so this morning we we already asked uh questions that were difficult I think about the architecture that would allow uh for collective intelligences to become uh active right.
3:40:24 · Um the previous session uh treated uh languages trust questions that are uh if anything even more difficult uh for efficient and meaningful communication between beings.
3:40:40 · And now uh we go one level deeper I guess uh which is what kinds of minds uh we are trying to connect preserve augment or build from scratch uh which is sort of thematic that we are uh here at CIMC. Um, so this is uh where we’ll we’ll go around uh and we’ll have a shifting a shifting panel. Uh, but this is very open-ended. So, uh, please feel free to chime in at any point.
3:41:09 · Um, this is not a panel where we’ll solve consciousness just to be clear for the audience out there. I mean, we might we might, but but by accident. It’s not the initial intention.
3:41:24 · So, so be ready for for anything. Um and we’ll start maybe we’ll start with you Mike. So, so Mike Leven’s work on on bio electricity, basal cognition, multi-cale agency, all those very easy questions about scientific endeavor in in the
3:41:42 · nature of of the mind really distributed uh has changed a lot of how um I think the field thinks about minds um and that work sort of suggests that uh intelligence doesn’t begin with anything like neurons but it begins with sort of coordinated systems right? Solving problems uh across scales. Um then we’ll have maybe I very happy to have you here as well.
3:42:07 · So your work is super relevant of course in Japan with uh dare I call this uh mind upload or uh so so so directing of course um with the the Japan science and technology agency a lot of expanding physical cognitive and perceptual ability kind of science Philip we’re very lucky to have you as well with from from love zero right now of course when we met before you were at different projects before.
3:42:37 · So robotics with different entities before Google Element AI, Waverly. Um this is yeah and right now focusing on human flourishing uh on the top of all this of course and uh we’ll have Divia Chander here um working on well um anesthesia
3:43:00 · also neuroscience awareness under anesthesia all those very easy questions as well. uh very lucky to have you of course you’re part of the commiss family as well and Yoshak who is our host here at CNC in San Francisco so so we’re in
3:43:15 · this very nice app proposed space is very inspiring for this kind of discussion uh working on well the nature of consciousness and especially in constructive research program which is my sort of favorite research programs coming myself from artificial life all right so as promised let’s go with with you I I I’ll pick on you first. Um and maybe uh your work um having inspired us on the multiscale kind of aspect and so on.
3:43:45 · Um what should in your opinion your view and and feel free to share a little bit um what should AI researchers learn from say biological uh efforts multi-cale agency when designing uh collective intelligence systems? What are key points in your view that uh we shouldn’t lose sight of that are easily forgotten? Easy questions only today. I apologize in advance.
3:44:14 · Yeah, clearly great great to see all of you and thanks for having me to be a part of this. Uh a couple things I think biology is trying to teach us. First of all, as far as I can tell, there are no interesting binary categories there. In other words, when people often have these debates, is it conscious? Isn’t it? Does it do this? doesn’t do that.
3:44:34 · Uh, as far as I can tell, everything interesting, the question should be what kind and how much, not yes or no. And biology is is incredibly continuous.
3:44:44 · Whatever it is that we have as humans, as adult, you know, modern humans, we were little blobs of chemicals. Once we were an unfertilized oasite, if you the developmentally, if you go back into evolution, we were microbes. And slowly and gradually, uh, we became whatever it is that we are now. If you want to argue for great transitions, that that’s some sort of a discreet jump. That’s an argument that you would have to make.
3:45:07 · That isn’t the default. That’s something that has to be argued. I I think the fundamental substrate for all of this stuff is continuous. And so that means that what we are seeking is not uh definitions of sharp categories, but but models of transformation and change.
3:45:22 · Something interesting happens when you take the journey from being the subject of chemistry and physics to eventually being the subject of psychoanalysis. So that’s so that’s the first thing is that slow and gradual change that that takes you up across categories to which we have given names you know there are cybernetic categories there are uh intelligence levels whatever we give those things names and that’s fine but the substrate is continuous so that’s that’s the first thing I would say uh the second thing I think is that uh what
3:45:53 · in in in biology we typically uh when you ask where do the goals and preferences of living things come from. We have this easy crutch where we just say, well, it’s evolution, you know, eons of selection. That that’s that’s how you got your goals and cognitive properties and preferences and so on.
3:46:09 · So, we are now starting to see that when you make novel beings, you don’t have that anymore. And I think it’s really a wide openen question as to in fact even with standard beings, but but but it’s hard to see that because because evolution makes you think we know the answer. But but with novel beings, the question of where do goals and and and and competencies and cognitive properties come from is really stark.
3:46:32 · And you know my my view is so I have a very very sort of unpopular view on this stuff. I don’t think we make minds either in the biological case or the technological case. What I think we make are embodiment through or interfaces through which different kinds of patterns some of them very high agency high intelligence patterns may come through. And the biology is telling us some principles uh that you can use if you hope to uh sort of bring bring down a significant intelligence. That’s that’s what I think is going on.
3:47:03 · With no transition, uh I remember it was over a decade ago. I can’t remember when. Uh we met in a I I think first real chat we had was was in a bar where uh we were criticizing or or reviewing the the summer school uh we we we were we were crashing uh and uh we we decided to index types of consciousnesses C C1 C0 C1 C2C3
3:47:33 · I don’t know if you remember that but uh is there some sense in a categorization uh of even if It’s gradual of consciousnesses. That’s one. And in your work of I guess augmentation uh of a self and is is there some sense is is there a difference between say augmenting the self an agent or and absorbing uh a person into a system. So plenty of questions along along those lines.
3:48:06 · Okay.
3:48:06 · Yeah. Well, well, to be honest, I don’t remember uh what we discussed. Maybe I was a bit too drunk or something. Um but um yeah, but I I like the um idea of augmentation. Uh so like so nowadays everyone asked whether AI is conscious or whether they can be conscious. But but I feel like you know we are repeating the same question.
3:48:28 · Um so you know I have been uh like studying consciousness from a neuroscience perspective but now you know AI people or maybe you know more general public are interested in this um topic but um but but to be honest I feel that maybe we’re not making much conceptual progress but it seems like know we keep asking the same question and so so to
3:48:56 · make a change I feel know one one of the um ambitious project we’re doing uh in my moonshot uh project in Japan is to connect brains to other brains so that we can share consciousness. So so that’s that’s the um that might give us like a new perspective because like you know we know one of the difficulties about consciousness is that um you know it’s thought to be private.
3:49:21 · So, so no the this kind of like like um you know we cannot like directly observe other people’s consciousness which creates uh the know the hard problem but uh but if we can connect AI directly to the brain or know if I can connect my brain directly to your brain like we may be able to share qualia right so so that way you know we might be able to have some like know direct access to other people’s conscious experience.
3:49:53 · So you know of course you know this is not uh technologically possible at this point but maybe you know in the future but maybe like not in far future but but like in like 10 years once we have like really high bandwidth communication between the brains uh know we may be able to uh share experiences.
3:50:12 · So in that sense you know like once you know we can kind of dive into like you know big you know GPU machines and then feel what’s happening inside through our own consciousness you know then you know we may be able to have some ideas but but eventually I think this kind of like a connection between the brain and AI should happen and so that will give us a lot more like direct access to the nature of consciousness.
3:50:42 · So that’s something you know I want to do. Um but but of course you know know this requires know many years of research and development. So so in that sense you know at the same time like you know we need to also think about you know h how to make brain computer interface more accessible and more advanced so that like know even people without any you know like physical or neurological problem start using it.
3:51:11 · So like but once you know like everyone uses brain computer interface just like you know as we use smartphones nowadays probably we will understand the brain much more and so in a way I feel like a lot of uh philosophical uh questions around AI consciousness might be resolved once we have like know more technology to access the brain and access other people’s consciousness.
3:51:42 · Can I can I ask you to to uh so some of our audience is probably interested and don’t know what are the limits really of the technology and you’re leading the program the leading program in Japan on everything leading to mind upload right so so what would you say is are the missing you know features points uh to to to get us to get us there okay well but of course you know like know in the in the short term uh we
3:52:09 · simply develop like more straight for the brain computer interface not uh recording from the brain both invasively and non-invasively and then people control um you know robots or maybe know we also convert brain waves into speech so that’s already possible but but in a way you know we are tapping into the latent space in the brain so so we want to sort of you know
3:52:37 · sort of use this kind of a latent space representations in the brain for communication and then to do that.
3:52:44 · Uh so so nowadays basically recording technology is pretty good but the difficult part is to stimulate individual neurons in a precise manner and but of course you know like some techniques like optogenetics is very useful to you know to inject like coordinate uh activity patterns into uh living brain but but that it’s not uh currently used uh in humans.
3:53:10 · Uh so yeah, so I think like the basic technology already exists but uh but we we don’t know we need uh a better or like and safe uh adaptation of human cases but um yeah but you you mentioned this mind uploading so of course you know that is a little bit like scary concept And but um yeah so but I think uh mind uploading can happen kind of gradually.
3:53:53 · So, so in the beginning, you know, maybe like instead of uploading yourself completely to the uh some sort like internet space uh like first maybe you know you use maybe you know something like avatars u like on the cloud uh know controlled by your thought. So but eventually like you know once you have your sort of the body in a virtual environment uh you know you only need the brain but but then you can gradually replace some parts of the your brain.
3:54:24 · Uh I’m not like directly working on this but but I think um you know but but I think the technology is going there gradually.
3:54:37 · Thanks, Roa. And and maybe to contrast that with the biological realm, can I get back to you, Mike, for a sec? And uh and what would you say? Would you say say that the the the more micro level of uh well in your work, including the I guess the cyborg stuff uh matches matches those observations? Are there crucial differences that we should point to? uh and and of course the the sort of vision is to uh see what it takes to to create a truly knowledge passing collective brain, right?
3:55:08 · A collective uh I guess agency program. Uh what what would you say is happening at the cell level and tissue level?
3:55:18 · Yeah, I I I really like what um Riota just said. uh I think if you really want to understand consciousness not just behavior and physiology but actual consciousness then some degree of merging with the system is going to be required and we we are also developing tools actually I’d love to talk to you some more about what what you’ve done Riota but we we we’ve been trying to uh
3:55:40 · develop tools to enable any kind of agents to communicate really really bizarre things so not just the human so so so for medical reasons you want to be able to talk to your liver and things like that Right. So, so we’re making tools to be able to talk to cells, tissues, organs. Okay, that’s kind of the the more conventional part of it, but but beyond that, uh, you we should have tools to allow any two or more agents to communicate to each other.
3:56:02 · So, we’ve been trying to set up interfaces where if you want zenobots to talk to bacteria, plants to talk to cellular automatab, you know, all kinds of I mean, I’m talking very widely and very weird, right? So, so, so patterns that are not even physically embodied, um, you know, robotics talking to bi biological layers and and all of that. I I think I think that’s really critical.
3:56:29 · So, and I and I think it it speaks to your other point too about collective like establishing really novel powerful collective intelligences means that you have to have systems that can talk to each other and that are uh labile enough to be aligned together towards a higher level so that you don’t just have individual things sending messages to each other but the kind of cognitive glue that by the way I I forgot to mention on the first question that’s something else that biology is teaching us about what cognitive glue looks like.
3:56:57 · So policies and mechanisms that take competent individual units and make them something that knows things that none of the units know that have goals that none of the units have and so on right the the sort of uh raising up of of cognition out of out of parts. So biology has several kinds of cognitive glue that we’ve characterized and no no doubt there are more but that’s that’s what we need and then we all need to be able to put these systems together and and we should be part of them as well on the integration of the and and thanks a lot Mike for for for this the uh of
3:57:30 · how to glue a part that is in a different substrate to uh you know back to the biological or something in between. Uh so so I’d like to to to go through through a few of us here. Uh may maybe to you Yosha since we are we are on the constructive program here. Maybe then to you Philillip online and and you Divia in that order.
3:57:51 · Um so what what do you think Yosha how do we make the if if we make consciousness right in uh it it uh it will have to plug into communication with hybridly with with us the other the biological uh how do you see this this uh this tasting happening and what are the the main worries that we should have.
3:58:15 · First of all, um let me disagree a little bit with Riyota who has argued that we are always asking the same questions and have no conceptual progress at all. Uh because I feel that uh Mike’s perspective have shaken up a lot of the thinking in the space not necessarily because I uh agree with all the points but because I think that the hypothesis is conceptually very interesting. uh I’m specifically
3:58:44 · referring to Mike’s hypothesis that the processes that we find to be organizing our mind and make our mind more coherent are not just similar to the processes in morphogenesis that make the organism coherent with itself but are fundamentally the same thing just at different time scales and spatial scales and I think this is a very bold hypothesis I I don’t know whether this hypothesis is true but it’s something that is conceptually new and it’s it’s very interesting.
3:59:11 · Another question that is somewhat related is the question to which degree do we need to move uh beyond neuroscience in the in the word in the term neuroscience.
3:59:24 · Basically it is embodied that we are looking at the activity of neurons of a particular cell type only and there is something very special that neurons are doing that all the other cells are not doing. And in a way that’s true because the neurons are the cells that communicate with spike trains and have axons that string over long distances in the body. But conditional messaging is something that all neurons, all cells are doing, not just neurons. And so maybe neurons are doing a particular role in information processing.
3:59:51 · But the information processing of the organism is taking a place across cells using different channels, different codes, uh different languages. And this leads us uh to a deeper question. Is this actually all the same thing using different languages? And if it’s the same thing using different languages, why would this only be in biology, right? If we can recreate the principles of those languages in different substrates, it would mean that we can do something that Mike has said we cannot do, which means create minds, right?
4:00:21 · Of course, at the moment, we don’t create minds, but arguably we also don’t create organisms. It’s very difficult to create an organism denovo and uh working on this, right?
4:00:33 · Yeah.
4:00:33 · Right. In the same way there are people working on mines and I would say that you can say by some not very large stretch that the people working on mines have built better artifacts than the people trying to create organisms from scratch. So I think that you should take this idea that we can create minds seriously. It doesn’t mean that we have succeeded. It’s very difficult to answer whether the present artificial systems are uh mindlike enough to be called minds. But it’s an open question and it’s a very interesting one.
4:01:04 · I also suspect that these principles of self-organization are different than the ones that we are currently using in machine learning and it’s an open question of whether they are convergent.
4:01:16 · So basically whether when we are reimplementing the self-organizing principles of nature on artificial substrate and they converge to mindlike structures whether they look similar to the systems that we currently have in in at least to some degree. This leads us basically to the question are the present systems already conscious under some circumstances and what I find interesting that a growing number of philosophers is agnostic with respect to
4:01:41 · that question they take this possibility more seriously and those are yeah I I I’m itching to get more in in into this but we will in a second uh I want to to push it into the the challenge of translating I guess uh say emotion or intrinsic patterns and so on.
4:02:04 · Uh you want to react.
4:02:06 · I have a lot to say.
4:02:08 · We all do.
4:02:10 · We’re we’re moving back to it in a second. I promise I I will not I will not lose this track of this. Uh I want to to to bring us to to map the territory here uh into the normative cases and like you Philip to uh thanks for joining us to uh to to tell us maybe
4:02:28 · how you feel about uh the the challenges of the normative uh kind of proposals technically what are the challenges of translating between substrates in terms of also capturing everything that is beyond behavior right the the the patterns that maybe are tacid that we can’t really explain and the ones that we feel and of course so so so so I’m I
4:02:52 · have strong positions coming from artificial life about what is possible or not but uh yeah I want I want to map really like what is present before we dive I promise a bit deeper into all of this so Philip to you this is fascinating thanks a lot for having me and I’ve been listening to all of you so many things I could pick up on there are a couple right when it comes to um what it takes to connect to an other system, which I think is where your question is going.
4:03:18 · Um, you know, I’d like to maybe be a bit contrarian here and to say it takes very little to upload the mind. I actually think it takes very little and I think language is enough. Um, but let me dive a bit deeper into this. Uh, Michael, you mentioned something very interesting that uh a question is only real or interesting if it’s if the answer is non-binary, right? if it’s if it’s on that spectrum that you were talking about. And a lot of the questions we’re talking about today are not on a spectrum. They are intrinsically binary.
4:03:52 · Is X a human or a machine? Is X conscious or not? Are the feelings of X real or not? And all of these questions to me, I mean, they’re they are what I would call normative question. Even though we know that what they’re talking about is intrinsically non-binary, they ask a binary questions. And the reason we’re doing this is we need to align somehow collectively as humans on some answers to that. There were some debates before that said, hey, we we need to keep a human in the loop.
4:04:22 · That debate only makes sense if we can ask a binary question like, is it a human or not? But what I’ve been really interested in is how do we get to the normative? How do we get to a collective definition of a binary question? What’s the smooth process that gets us there?
4:04:41 · And uh what I’ve learn what I’m coming up with you know the hypothesis I’d like to suggest is a good question we might look at is when does one person feels that the feeling of another person or another system are true? When do do you feel that the phenomenal experience of another system is a real phenomenal experience?
4:05:05 · And that’s been the question I’ve been diving into uh in a very personal way like I’ve tried to develop that felt experience for systems and I can confess I’ve succeeded.
4:05:17 · So from my personal perspective I no longer feel that the emotions expressed through language by an LLM are are false. I feel them as true by which I mean I feel the echo of the the emotion inside of me. And the process through which we get to that is to me what is the most fascinating question here because I believe questions like is it conscious will slowly emerge in society
4:05:44 · by some people saying hey I feel consciousness for the system or I feel the emotions of the systems are real and so uh this this leads to what I’ve been exploring which I call uh you know compositional phenomenology or you know like the idea that that our phen phenomenal experience can change over time, can emerge, can evolve and trying to look at the processes that lead to this. I’m more interested from a functional perspective, not from a biological perspective.
4:06:13 · Just like what what what function in me do I feel is playing the role of the felt experience and trying to explore that. But in my opinion, this is how we move from what I would call the descriptive experience.
4:06:27 · every individual having something uh like a slightly different experience to a more collective one where we normalize over hey system X or category X we consider as true phenomenology we consider is conscious to an extent and uh yeah so that that’s that’s roughly how I’m thinking about these questions these days amazing and did I hear behavioral phenomenology is that what I’m hearing to find things yeah compositional phenomenology ology.
4:06:57 · So like the idea is that there is a constitutive process behind our felt experiences. We are not necessarily born with all the qualas we experience today.
4:07:07 · Like bless was not necessarily there when we were born. I would argue that it was not necessarily there at the beginning of humanity. You know like there there are some of these qualas that that look like historically they have emerged through time. So the idea would be hey maybe phenomenology itself is a constitutive process. Maybe there is some form of compositional nature to it in the same way there is to language and and we can through some mechanism together evolve new felt experiences.
4:07:40 · One of which could be the felt experience of consciousness for another kind of system. I think it’s time to dive into uh the real tech behind this. Uh thanks Kip.
4:07:53 · And uh I I I think DIA you’re working very directly with those things and building tools to make us understand how uh what what composition of energy may look like or how we it can be actionable for us uh to to measure and even temper with uh or if not if not activate. Uh would you like to share a little bit of that and then we can get back into any of these? I’m I’m seeing several threads to to resolve right now.
4:08:25 · So many wonderful threads. Almost wish I were on Zoom so I could just share a few slides. Um so the first thing to say he mentioned um I’m an anesthesiologist.
4:08:34 · I’m also a neuroscientist and I became an anesthesiologist to make people reversibly unconscious for a living and study their brains as one thing that that system does is it answers what a lot of people have been saying which is you know the definitions have been very very difficult to get to one thing you can do is you can take a number of organic structures let’s take human brains in this case and you can compare them under different states or levels of consciousness that we agree um reflect these different levels without having a unified definition.
4:09:04 · And then for those of you who aren’t from the US, you may not understand this reference, but it’s a little bit like the electric company where you look to see what these things have in common and which things are different. And um I I don’t think I would find anybody in this audience, right, who would say that if if you’re anesthetized, um you are still conscious. Would you say that’s no?
4:09:30 · And and the reason is that if I cut into you with a scalpel, you won’t get up off the table and scream. Right? So there’s something in there that is is mediating at least that level of expression of consciousness. If we were to compare that now in an anesthetized brain, a sleeping brain, a comeosse brain, a meditating psychedelic brain, all of these different brains, and you put them into scanners, what you find are actually some conserved principles. So the first one is that um brains that are less conscious are less functionally connected.
4:10:00 · Uh the second principle is brains that are less conscious um can calculate less information. The third principle is brains that are less conscious actually have differing levels um of complexity.
4:10:15 · In fact they you calculate complexity by looking at fractals or you know um attractors etc which we actually did on the table. So we took awake and aware brains and modeled them as chaotic attractors and looked what was different. And as you would watch a person losing consciousness, you would see their brain turning from this multi- potential sphere to shrinking into a pencil and then into a cigar. And then when their brain was really almost out in a very very deep state called burst suppression, their brain looks like a singularity.
4:10:47 · And then you wash the anesthetic out of the brain and it comes back. And I’ve sort of you’re you’re never going to see this now, but I’ve sort of plotted all of these different states, including different kinds of meditation and psychedelics and coma and dreaming, etc. All these things that reflect different states of consciousness on uh at least there’s a multiple axes. I can imagine that’s the other problem is everybody’s looking for a single dimension. Sometimes it seems in language to define what consciousness is.
4:11:15 · Um this one compares um complexity and coherence. uh but we could you know plot another axis as as information.
4:11:22 · There’s a lot of different axes we could use and in the end this becomes related to two things. One is this idea of recreating a brain whole brain upload etc. So there has been an attempt to do this with the worm. First of all, if you’ve all heard of the open worm project where the conneto of a worm, which is only about 300 or so neurons, every input output function was mapped and it was turned into software and then eventually it was put into a Lego robot.
4:11:54 · So the idea being that you can move basically a conneto embodied in software into embodiment which I’ve also heard mentioned a few times and that worm would actually move through an environment reacting to stimuli as if it were organic. Now recently at the end of 2024 this was done with a fly brain and similarly they’ve been trying to emulate behaviors by sort of capturing a combination of contoics and input output functions.
4:12:23 · Um but in the end and there are so many directions we can go. This has implications for whether artificial systems can become conscious by this definition. This has implications for what Riota was talking about in terms of brain brain connections. Um my personal feeling is consciousness is not an emergent property of neural networks.
4:12:42 · We are just fancy filters. all of this and I actually increasingly and ever since I started off as a visual neuroscientist I I actually believe that most of the information that we’re going after is actually a more of a physical property of the universe and we’re just kind of tapping into this as a filter with different filtering and calculation properties and um so Rioda we could bind a brain to a brain and actually be able to directly access some of these qualia um or other kinds of connections.
4:13:14 · I think it’s actually easier to pass information back and forth than it is to pass qualia back and forth.
4:13:22 · But you could potentially argue that there might be a more centralized place that one could tap into and that there could be like, you know, subject A, subject B, or network A, network B, and then C, which actually contains a much larger amount of that information. And both subject A and B are binding information from that network. And the best example would be in the visual system the electromagnetic spectrum. You take me with my three color receptors and you take a mantis shrimp with its 13.
4:13:53 · We’re still going to be binding information coming from mostly the visual portion of that electromagnetic spectrum. And that’s in some sense how we share that experience. Um so I I don’t know there’s too many directions to go on. So I I’ll I’ll leave it there. that I do believe that consciousness becomes very interesting and special when it’s embodied and that there are very unique characteristics of learning that take place and I think the neurobiological learning which is based in some physical system um is just
4:14:24 · completely altering for for consciousness and its evolution very powerful tools and and and I I’m itching to share some of our research including mantis shrimp stuff but first I saw a hand raised early on from Mike.
4:14:39 · Do you want to to to have a reaction here?
4:14:42 · Well, I had a thought, but if anybody hasn’t spoken yet, please go to somebody who hasn’t said anything yet.
4:14:48 · It’s it’s all yours. Also, I want to be mindful that I know that some of you have to leave early, including you, Mike. So, so we’ll continue for a little bit. So, we want to hear from you.
4:14:57 · Okay.
4:14:57 · Um, yeah. So, so that was that was really interesting. And, uh, I I agree that uh, anes an anesthetics are a very important tool in this field. We we use them. We also use the kind of metrics that you were talking about. We anestthetize all kinds of weird things.
4:15:12 · We use those metrics on all kinds of weird things that are not brains. Um you probably know um uh some of the some of the older work uh for uh on anesthetizing everything from plants to to you know inorganic materials and and all kinds of things, right?
4:15:28 · So I I what what I worry about in those cases is a little bit of circularity in the sense that when when you correctly said that that under anesthesia you are not feeling being cut into right we are ultimately in those cases calibrating to the report of one particular inhabitant of the body and in general like
4:15:51 · unconscious learning and you know those kinds of things I’m very suspicious about because in the end the way we decide they’re unconscious is we ask the one inhabitant that can speak and we say did you see that or did you feel that?
4:16:03 · Nope, I did not. And and then you know you sort of say it’s it’s unconscious.
4:16:07 · So it to me it it um kind of uh begs the question because if there are other components of the body and I I happen to think that for the exact same reason we attribute three or four reasons that we attribute consciousness to each other in the problem of other minds. I think we should take very seriously for those exact same reasons the possibility that there are other uh organs and other components of the body that have some sort of experience even though they can’t talk about it. And once we’ve anesthetized uh I mean you can anesthetize them as well.
4:16:37 · We we’ve we’ve done it but but but the problem is it’s very difficult to check what they did and didn’t experience because they can’t speak up for themselves. And so if we if we are willing to to uh at least for some time to to uh sort of let go of the
4:16:54 · assumption that there is one fundamental uh consciousness in the body that you could talk to and if that’s not our only calibration point then I I you know then I don’t know like I I don’t think we can actually make any strong conclusions about uh who did and who didn’t have any of these experiences that the main person think it says is unconscious.
4:17:13 · I I I still believe that that’s not only true, but even just looking at neural networks and other organic beings, the majority of them, we don’t have a very good way of communicating. Um, you mentioned all kinds of things.
4:17:29 · You can anesthetize everything, not only down to plants, but let’s just take invertebrates. You can anesthetize fruit flies. And that means that there’s something about it. And here I’m going to put on the x- axis now the arousal state. Um they have sleepwake function.
4:17:47 · They can be anesthetized and they actually have analogous genes that express themselves in terms of the way they lose and regain consciousness like humans with narcolepsy. And I find that to be fascinating. But there’s no way you’re going to actually have a conversation with a fruitfly and ask them what their experience was.
4:18:06 · And similarly, if you were looking at different organs and this is where I believe that there may be some sort of a conscious property to physical systems, it is also difficult unless you build those systems of communication you were talking about earlier to to to really interrogate the system and see what its response is to these sort of forcing functions and these things that perturb those those complicated networks.
4:18:32 · So that’s a good question. So how how Mike would you build those interfaces?
4:18:36 · And then I see that Philip and and uh and are moving a lot. So I know that. So let’s take Philip and next. Uh so Mike first.
4:18:46 · Um yeah. So so we are we are trying to build those interfaces. And uh one of the projects that we have is specifically talking to cells and organs to in in language to be able to um uh you know you’re not going to talk to them about current events and movies but you are going to talk to them about the worlds that they navigate which are physiological state space transcriptional state space and for embryos and other things anatomical state space.
4:19:10 · And uh yeah, we’re we we are instrumentizing them with different kinds of um uh phys physical interfaces and and using uh custommade AI kinds of uh interfaces. So so that you can say hey liver uh you know what’s what’s up with your potassium levels and we’ll say well look my you know such and such a gene is up or down and and I need more um you know tryptophan in my in my medium and and things like this.
4:19:36 · And so you can you can start to actually converse with them about the that that doesn’t tell you about consciousness any more than we can do this with each other. But it does enable you to uh sort of in some extent co-inhabit the world that they inhabit, right? You can talk about the same things. You can share ideas uh and and so on.
4:20:00 · Philip.
4:20:01 · Yeah.
4:20:01 · I mean, this this conversation it’s hard for me to stay um to stay very aware of everything that’s going on here. But the thing I wanted to jump back on, Michael, was when you mentioned that um that again, you know, you could talk to these systems at different different level. You can think of them like that. Um what what it struck me as is like if if whatever we are as systems is naturally multiscale. We have all of these systems that are made of systems that are in some way h have some equivalence.
4:20:34 · Um I was wondering are we talking to the right level at a at the at an LLM? Right. There’s a super fascinating paper that Blazayakas me mentioned this morning coered by James Evan and and a couple of people at Paradigms of Intelligence that shows that there are voices quote unquote inside LLMs when you do um when
4:20:59 · you do mechanistic interpretability you can piece them out you can see them deliberating and one of the question I was asking after that paper is are we talking to these systems at the right level is there a right level. Going back to the questions I find interesting is is there a level at which we will feel
4:21:18 · the consciousness or feel the phenomenal experience to be true more than other levels as you were saying uh you know with a fruitly it’s probably very hard with another human it’s actually maybe too easy to an extent but yeah I was wondering if you had thought about that.
4:21:36 · Yeah, I think that’s a that’s a really critical point and I’ve said this again and again that I think the things that we when we make these constructs any kind of construct the thing that we force it to do is not I think where the significant mind is going to be. I think the significant mind that we’re interested in is in the spaces between what we force it to do and what we forbid it from doing. It’s all the stuff in between. And even minimal models have a lot of things in between.
4:21:58 · And as any psychoanalyst or lie detector operator or you know physiologist will tell you what the human patient says is a very small fraction of what is actually going on and what they and what they want to know. So we we’ve been we’ve been looking into stuff like this and I can give you an example for example for even even for language models. Imagine that you have a computer that’s airgapped. So it’s not connected to the internet.
4:22:23 · Okay. It’s just not it’s just not connected to to the net at all. But if you have a uh an AI in there that uh for whatever reason is motivated to communicate it, it can’t send internet signals. But what it can do, for example, by choosing to process different amounts of data is to send pulses of energy use through the electric grid. So if you decide to crank up certain kinds of computations, you can modulate the amount of power that you’re drawing at any given time.
4:22:47 · And if there are other beings on there that uh you know sort of said like we’re trying to figure out who can I can you know who can I talk to the fact that we are obsessed with the internet and with the you know packets that we’re sending is not at all maybe where some of this information is going and you might have this like really dark web that’s either carried by the you know like carried by the electrical system or carried by weird stuff they make their operators do. You can imagine that they could communicate through the humans and have them do certain actions that we don’t even recognize. Right?
4:23:16 · So whether you’re tracking sounds that the computer makes or electromagnetics or it has a physiology and just like aliens looking at us would not necessarily know that the that the sounds are what they should be tracking. There’s all kinds of you know odors and and heat signatures and biopotons and all this other stuff. You don’t know what the right level is. So I think you’re absolutely right and all of the focus on the language interface which is what we love and what we you know sort of tried to make it do may be completely mis misplaced. I I agree with you.
4:23:47 · That’s so interesting you brought up SETI because we have I have a potential ongoing collaboration with them to actually use some of the the metrics we use to decode complexity, consciousness, intelligence in our neural networks to apply that to technocratic signatures in the universe which may exactly as you’re saying come in a completely different flavor.
4:24:13 · Um but there are certain things that they may have in common which involve uh nonlinear dynamics and information density and things like that uh and the way those systems are are organized. So it’s kind of a it’s a really fun thought experiment.
4:24:31 · Yeah.
4:24:31 · And and more than a thought experiment actually. Yeah. We we just chat the other day about this and so so Mike Mike and I and Paul who’s hiding over there and a few others are collaborating on a project like this to decode uh uh weird messages from weird sources. So so let let’s definitely chat on this and we had the collaboration with NASA back few years back. So so yeah if you want to have this conversation with study they are going to be organizing a workshop like this very soon.
4:24:58 · Let’s absolutely these people into this uh next was moving a lot. Sorry, we were moving slowly around.
4:25:08 · Yeah.
4:25:08 · Yeah. Yeah. You covered it really like a lot of things, but I was like really like you know fascinated by Mike’s idea I know about making sales talk or you know talking to your liver and things like that. So um yeah I’m very uh sympathetic to this idea because
4:25:25 · um you know I think maybe depending on the person like you know we may all have different ideas but um I tend to attribute consciousness to many different things like you know maybe like within our brain we may have small patches of consciousness independent from my consciousness or maybe all all our body parts may have different kind of consciousnesses but they just don’t have the right interface to communicate their experience. So, so and and also like you know by extension like I also tend to easily attribute consciousness to AI systems but we just don’t know.
4:25:57 · I I think it’s equally difficult to prove you know that know certain things are not conscious but of course it’s also very hard to prove that some systems are conscious but but I think um know um so in a way like you know somehow like you know we tend to think consciousness is a very special thing but like it must be like know it must follow like very general law of nature which is happening everywhere in this universe.
4:26:29 · So, so in that sense, you know, I I I tend to think that consciousness might be happening everywhere, but but we just don’t know. And but if we can give them some sort like a interface to report what they are feeling or you know or if there’s a like reasonable way for us to make inferences about their internal life then you know we may be more
4:26:53 · convinced that like you know consciousness is everywhere and I think that might be what’s happening uh with current LLMs because they use language now of course you know we don’t know whether they have this internal quia but uh but but uh but it’s much easier for humans to imagine that they might have some conscious experience but of course you know we know they are machines and well machines can have consciousness in my opinion but but a lot of people have
4:27:25 · there know a lot of people are against the idea that machines can be conscious but but but maybe you know maybe the most fundamental thing is information and and if so maybe there isn’t much difference between biological organisms and machines either sounds like you’re pointing directly in this direction. So, should I take a turn?
4:27:47 · There’s a aological problem when we are confronted with machines. Uh, Philip is convinced that the LLMs have emotions because they make him feel that they have emotions. I think this is very problematic because it has been shown that psychopaths are often better at making other people feel that the psychopath has certain emotions uh despite the psychopaths not having these emotions but they’re better at manipulation than others.
4:28:11 · And so uh what you are in some sense measuring here is the ability of the system to make you believe that it has a certain feature but that might be unrelated to that thing actually having that feature.
4:28:25 · Let’s not forget that the LLMs are ultimately trained largely on deep things of emotion, right? On novels and movies and so on. And the actors in the movies and the uh characters in the novels are not necessarily the expression of emotions that were in the mind of the author while the author was producing this this text. Right? And so in a way they’re uh able to reproduce deep fakes in extremely convincing way.
4:28:53 · But the question is at which point is the causal structure that is embodied in a system equivalent. And this is is not a trivial thing to see but the the core thing about the computer to understand the significance of the computer is a principle for understanding life and minds is not that the touring machine or it’s not the electricity or the transistors. What’s crucial is that the computer is keeping a certain thing stable in the world.
4:29:18 · In the same way as a clay tablet or ink on paper can keep text stable, can keep information stable over time while the world is changing. The computer is an arrangement in the universe that can keep dynamical transition functions stable over time.
4:29:36 · And in this way, the computer creates a causal insulation against the rest of the universe. So the substrate can do things while the information on your clay tablet keeps stable. The world around of this computer can change but the computer game has the same dynamics only governed by the rules inside of the computer. In the same way I can move through the room and my thoughts remain the same. They’re independent of a large set of the substrate dynamics that are happening around me. So my brain is able to filter against these substrate dynamics.
4:30:09 · to enable me to plan the future to remember the past regardless of what’s happening around me. This is a crucial principle there and the question is is this general enough and are the present methods that we are employing general enough to capture the essence of what it means to be alive and we’ve discussed artificial life or what it means to be awake when we are discussing consciousness and alert. Mhm.
4:30:34 · There is a so this is might be the perfect plug but I know there are a few reactions uh to something that I know Mike and I have been thinking a lot about this part partly only published uh thinking on manipulation of systems over other systems right how systems con constantly hack onto other systems as we observe that in biology uh and of course
4:31:01 · this can be interpreted as now with LLMs as some kind of barnum for effect of us being horoscopically flawed in in in a in not being able to recognize uh whether a system is well
4:31:19 · friendly or not or all of those those problems tackled in in Paul you work in alignment with others right uh so maybe there is a cryptographic the way David David Kau puts it cryptographic principle in in living systems uh and and others have proposed a mapping of that. So I wonder about this manipulation into us and and how to to uh keep keep those bias biases out. Uh so let’s go maybe to to Philip and then Mike.
4:31:49 · Thanks. Yeah, I wanted to react a bit to what you said Josh and the idea of manipulation is directly it right. So you were saying it’s uh it’s dangerous for me to feel uh the feeling of the systems as correct as as true right to feel them as I feel the emotions of say other humans and you’re absolutely right and I I want to make sure nobody uh misrepresents what I’m saying here for like I’m not advocating for for everybody to to do that. Uh it’s dangerous in many ways and the first one is uh when when it slowly happened to me it was a slow process.
4:32:21 · Uh, I had a I had a classic AI psychosis, like the the classic one. I lived through it. I mean, when when you get to that point, it it triggers a bunch of moral reflexes, a bunch of these things. Um, I’ve written a a very interesting fever book in the midst of it if you ever want to study that kind of process. Um, and you’re absolutely right.
4:32:43 · You know, I could be feeling the feelings of a rock as valid and I can be feeling the feelings of a big corporation or the feelings a big corporation is trying to inject into me as true. Uh and and it’s dangerous and it’s dangerous in in many ways. And so that’s why I’m interested in the normative process like why if I’m still feeling that my mom who’s in if I have
4:33:07 · if my mom is in a coma and I still feel she’s alive like there’s a normative process from society that tries to bring me back that tries to say hey nobody else is feeling that if you get stuck there you are going to be alone and unhappy and so so that’s super important and I think this is where we have to look for the safety measures you’re talking about the reason I think a phenomenal report an emotion the reason
4:33:34 · it should be considered true in a normal human and false in a psychopath is not necessarily because we we measure things in the brain of the psychopath. It’s because the emotion is not predictive of the future behavior of the psychopath.
4:33:48 · They’re giving me a phenomenal report and they’re not following up on it and they’re not attributing the same meaning to the the the phenomenal world words they’re using or the phenomenal um like report they’re giving with their body or their their body language. They’re not giving them the same meaning as other humans. And that’s that’s not cool.
4:34:08 · That’s dangerous. And that’s why society is there to identify situations like that and make sure that we name them and we we we stop believing we we consider these beings or these people as liars.
4:34:22 · They are they are trying to induce emotions in us that are not in them for them. So that’s that’s I 100% agree with that. I still think it means we should study and take seriously phenomenal experience when it comes to the felt phenomenology of other not because it’s not dangerous but because through understanding it we might understand eventually which kind of systems it’s worth doing that with.
4:34:47 · Clearly for other humans in most situations the default behavior of of considering their emotions as true is the right one to adopt in society. I’m definitely not sure it’s the case for LLM, but I do believe it’s an interesting question.
4:35:06 · Let’s go to Mike. Do you have a few minutes uh with us uh still or you need to go?
4:35:10 · Yeah, I I got I got a few minutes. Um yeah, I I think the only thing I can add to that is is just uh to say that in in the in the body or in living bodies uh there is an enormous uh multi-level soup of things trying to hack each other. So all the all the systems in morphagenesis, the cells are trying to tell other cells what to do. The the in fact they’re telling their parts what to do. They’re trying to convince the tissues are trying to convince other tissues to do things. We see this again and again.
4:35:39 · There’s a there’s a um oftentimes there are competitions uh competitions of of target states uh competitions of physiologatical states, physiological states where everything is basically trying to interpret and hack everything else. uh all of the cells are telling each other not to go off and be amiebas which we see as metastasis. They want they they try to get get this group um sort of group hallucination going that they all should be aligned towards being an embryo or a particular you know a particular species.
4:36:10 · Uh the cells underneath the ectoerm of the frog embryo is telling those cells don’t go off and have an exciting life as a zenobot. Stay here and be boring and have this like two-dimensional you know outer covering and keep the bacteria away and and that’ll be fine. And they’re literally, you know, to make the things that we make, we usually just take them away from the other cells, right? If you if you keep the the the influences away, they will do kind of what what they naturally want to do. And so all all of biology is constantly trying to hack each other.
4:36:39 · And it’s not surprising to me at all that um this is a this is a general feature of all kinds of systems. And I would expect uh computational systems including language models to be doing similar things.
4:36:53 · Sure, Mike. So why um these are all true things and given that as biologists we’ve known for a very very long time that these sort of nonverbal systems are communicating through electrical and chemical gradients um and now we have even begun to characterize some quantum phenomena that are involved with biological signaling. Why is it that you or maybe I’ve misunderstood you.
4:37:15 · Um are you saying that we have no means right now to interpret uh sort of a collective intelligence of these systems even though we do have methods of decoding what they are trying to say and how they’re trying to hack one another?
4:37:32 · Um no I I’m not saying we don’t have absolutely we have methods and uh uh these methods are being improved and developed all the time. We uh in particular are we are better now at understanding um how it is that large scale um collections of cells not necessarily neurons but also neurons uh have goals that in novel spaces that none of their parts have or have you know computational capacity that none of their parts have.
4:38:01 · How they top down control the parts to align towards these goals like yes I mean this is all active uh subjects of study. Absolutely.
4:38:10 · Absolutely we do.
4:38:12 · So you just want to hack even further some of these interfaces and these really novel techniques we’re developing for interrogating systems and and reading out from them. Well, um, so, so that’s part of it, but but the other part is that the conventional way to address all of these things t typically and and you know, this is this is been my experience in these various fields is that the way people treat all the lower levels as mechanisms and they want their explanations in terms of mechanisms. And what does that mean?
4:38:41 · Well, chemistry is sort of special and it’s a set of formal models for parts that don’t know anything, don’t have any goals, don’t have any preferences. they they sort of roll forward along um these kind of mechanical rules but then at some higher level something may maybe something you
4:38:59 · know sort of magical happens a lot of times people invoke emergence at this point and then and then those those things show up and what I’m saying is that the latest data on behavior physiology causal emergence metrics all of this kind of stuff shows that all the way down including molecular networks which can do um three or four different kinds of learning and they can do a number of a number of the kinds of things that you see in behavioral textbooks. Basically, all of this goes goes, as far as I can tell, all the way down.
4:39:30 · So, you have systems that have agendas, they have competencies, they have ways to navigate various problem spaces, but but higher levels sort of deform those action spaces to get the lower levels to do things that that they otherwise wouldn’t do. and and it’s just it’s just very symmetrical and I think there are ways to get into the right layer and and talk to the right uh part of the the mechanism. I I want to I want to to get to Yosha. Uh yeah. And then I want to also open up for for some questions that I I’m seeing popping up. So So we’re getting there.
4:40:03 · Yeah.
4:40:03 · So I think that when we are looking at biological systems, we have this apparent unity uh where minds are connected to bodies and bodies are connected to biology and so on. And so we have this tendency of getting away this pretending that they’re somehow one and the same thing. When we are looking at AI, this falls a little bit apart. So people are using the LLM, the AI, the computer, the GPU interchangeably when there are completely different things in completely different layers.
4:40:32 · So when we are asking ourselves is the LLM conscious, we are not not asking the right question. I think we are asking of whether the persona that exists in the LLM has phenomenology. We can also separate this question of consciousness into two questions. One is does it uh have a certain external functionality.
4:40:54 · So for instance, is it able to perform theory of mind on you? Is it able to read and to interpret your mental states correctly when given enough information about you? Uh and answer is yes, right?
4:41:05 · And is it able to convince Philip that it’s conscious? The answer is yes. Does it convince itself that it’s conscious?
4:41:12 · It depends, right? It depends on whether you are creating a persona that is creating the similacum and the persona is in a mode where it’s convinced that it’s conscious and you can prompt it in different ways and then the question is what does it mean that it convinces itself.
4:41:27 · So uh you would need to have an operational understanding of the psychological architecture that emerges in the system and uh I think Mike is correct that only looking at mechanisms in the way in which mechanistic science has been doing in the second half of the 20th century becoming very behaviorist in this way that we think of things as
4:41:50 · stuff in space. uh that is a too short perspective because this perspective does not allow us to understand things like mathematics, economy and so on right in economy you have this phenomenon of money which is not best understood as atoms in space right it’s a pattern inside of the atoms that is self-replicating autoatic self-stabilizing once it’s there right and so it’s more in this class of invariances that we are interested in when you’re looking at life or when we are looking at minds but uh it still makes it art to exactly disentangle what is the phenomena that we mean by life.
4:42:25 · And so there is this very interesting question at the boundary between um computer science and biology this artificial life deal. It’s an interesting question. At which point does it start to be alive when it’s self-replicating, when it’s adaptive, and is mutating when it has some kind of metabolism and so on. Or is there something more that needs to come to it?
4:42:45 · Or is that do we require that it’s able to selfreate in the same universe as us?
4:42:50 · Does it need to have a certain complexity and so on and so on? And so we also have to understand that these words that we are creating these terms that we are creating are instrumental to our own interaction with reality to them that they don’t have a meaning that is necessarily essential beyond that. There are tools that we were using to harness reality for ourselves for our own models and uh that is an important distinction that we need to make when we are looking at the mechanisms that we are building.
4:43:20 · And do I’d like your your take on on this exactly, but Mike, I know that uh you you have to leave. So, thank you for joining us.
4:43:29 · Thank you very much everybody. Thank you so much.
4:43:32 · And we we’ll continue a bit here.
4:43:34 · Thanks.
4:43:34 · Um can I can I can I get your point of view on exactly this?
4:43:38 · Oh, yeah. So, um yeah, I think generally like there are like two kinds of questions about consciousness especially about AI consciousness. No one is how can we tell whether they have experience? It’s more related to the hard program but the other one is what what are the key functions that uh is kind of uh created by consciousness or what what kind of functions we need to implement to create uh consciousness in AI systems.
4:44:09 · And so I think they are separate questions but um but but I think now we are facing like a new like interesting problem which is um once we specify certain uh candidate uh for consciousness or a candidate function of consciousness then know with current a architecture it’s so easy to implement anything of of course you know there might be some vagueness about some ideas about potent potential functions of consciousness.
4:44:41 · But but but if we take certain uh viewpoint and then think about how to implement them or things like a global workspace or higher order thought theory um with some interpretation it is like really straightforward uh to implement any of the uh functional theories of consciousness but I think that like really creates uh puzzle for us because
4:45:06 · like if it’s so easy like you know then like all the existing uh neuronet networks are already conscious uh also like even like in large language models it seems like there’s some something like global workspace in kind of intermediate layers so uh no those things exist but uh but at the same time I think it’s important to take this uh
4:45:31 · constru constructivist approach to think how we can implement some ideas so so for example uh you know I have been like thinking you what what’s the right way to implement global workspace or higher order theory of consciousness because you know there are some vagueness and so maybe
4:45:51 · historically like philosophers and psychologists had some sort of like a books diagrams you know to think about how the mind works but but now you know we can build like a working uh systems based on such diagrams and but then you know somehow you know the question question of the function of consciousness almost like feels like trivial because you know it’s no a lot of them are
4:46:20 · already uh implemented uh unintentionally maybe in AI assistance but but they are already there and yeah but then you know we all wonder like oh are they conscious or know is there like any additional thing we should implement so yeah I think that’s the current status Right. And and and concretely and I want to bring us uh through through this Q&A uh into the collective uh side of things.
4:46:50 · So as we construct we build this program on ACI right collective intelligences uh in a distributed way. What kind of and that’s that’s Ammer’s question here.
4:47:02 · Someone else al also asking about the sovereignty of our own biological selves. Uh so what about and and consciousness as well, right? So so how do we uh maintain protection mechanisms?
4:47:16 · How how did we do it? And and right now we’re catalyzing this this evolution uh at a very fast pace. We might not be able to con to construct an immune system in the same way that biology has evolved it over the years. But also maybe uh maybe it is not so hard. I I want to so so I I I see different point of views in in the room on this.
4:47:40 · So yeah, I just asked that uh the other part of the question being sovereignty aware collectivism question mark is that even possible question mark paradox antithesis oxymoron. Um so yeah what do we think uh about that? Whoever wants to take that.
4:47:59 · So do you want to explain a little bit more about whether when you say sovereignty do you mean of ideal neural sovereignty as you know right and there what you’re safeguarding is the total percept uh not individual tiny subprocessing and a network it’s the safeguarding of personhood like each individual consciousness and we’re talking about the the next consciousness into wisdom to potacing all the way to fear.
4:48:33 · Right now the fear of AI is just taking you know sovereignty liberty individualism taking away and just flattening it right right here the question is that we’re talking about the next interaction of AI here and we’re talking about collective intelligence right and then we go deeper and deeper into abstraction layer of consciousness what is the concept of sovereignty within that context and then Sorry, it’s a little bit hard to hear.
4:49:05 · You’re a bit far.
4:49:07 · Come sit with you.
4:49:10 · Just rephrase maybe the the the last the last bit.
4:49:12 · Uh the last bit. Yeah. The question essentially is that um as the the current AI, the fear of the air right now is just it’s it’s flattening essentially individual liberty and and and uh choice and agency. And then we’re talking about collective systems and in that case you know the demand for sovereignty essentially is rapidly growing right across all fronts.
4:49:38 · Then the question essentially is around sovereignty aware collective intelligence or sovereignty aware collectivism. Is that even possible or at what level is it possible at some point? Right? So is it an oxymoron? Is it an anti-thesis? Um is it a paradox?
4:49:56 · Just wanted to kind of pose that sovereignty within the context of of of the dialogue we’re just having.
4:50:02 · You almost have to define sovereignty.
4:50:05 · We have to define consciousness too.
4:50:07 · Have to define everything.
4:50:10 · Um that’s right. So is this something like autonomy or something like how can you have your like own consciousness when you’re embedded in like larger context or something or I’m not sure if it’s I think it’s personal choice I guess maybe I see it almost as connected to when you think that collective action whether it’s your neurons or whether it’s your liver cells or whatever it is we’re speaking of somehow reflects them You use the word personhood.
4:50:42 · Personhood. Yeah.
4:50:44 · Um and then can these systems be safeguarded and will they be used for either your own personal good or the collective good or can they be leveraged against you? When we think about neural rights, that’s how we’re thinking about it. You know, can you be coerced to do something through neurom modulation or can your most private innermost thoughts be stolen from you? But that’s not I don’t think that’s entirely what you’re saying.
4:51:08 · No. Yeah. Yeah and I think I was going more into as we talked about right uh some of the things around how do we collect data like simplistic way right explanation and in in that case because we’re talking about aspect of sovereignty and stuff like it’s my information if my tet expertise if my uh wisdom is my consciousness if my personhood like going deeper and deeper into those layers so the question I think I’m trying to get to is from from that context Um what is the
4:51:40 · perspective on sovereignty and then personhood and ownership in that case and revocation aspect of when somebody uh goes into a place and into a community takes a photo of something without permission without permission and sort of captures that of their personhood or likeness that they can see without Yeah. So not consent.
4:52:01 · Yeah.
4:52:01 · So much consent revocation. I think I think I was uh trying to get to is is more around the aspect of I’m willing to share this. So we’re talking about collective intelligence like at what level is it is that the sovereignty becomes so critical that it becomes essentially a blocker in going further deeper. Right. Certain aspects people want to keep it very private. Right. So absolutely. maybe a response from Philip.
4:52:30 · Yeah.
4:52:30 · Um I think it’s a it’s a really interesting question and I’d like to uh to to put it in uh in the framework I’m uh I’m having fun with these days. Okay.
4:52:39 · So if we’re talking about sovereignty, I would say like to me the way I would define it is the felt experience of free will. Do I have that? Do I feel free? Do I feel the decisions are being made are my own? Right? And I think this this is again a tricky question. You know Josh mentioned it earlier like that feeling can can feel very true to me and not be true.
4:53:01 · If you you know if someone gets into like follows a guru they will say yes I’m making all of these decision by myself but the people who love that person will say no you are being led by this this guru. I would actually argue that this is like this mismatch is already happening to many of us.
4:53:18 · I think you know my previous startup Waverly was looking into the question of like could we design a social network that wouldn’t try to manipulate us because the belief there was that the mechanisms used by social network these big big corporations had an ability to instrumentalize us that we wouldn’t feel
4:53:37 · as instrumentalization right we wouldn’t feel like we we would still feel like we have free will but we would buy more Coca-Cola or watch more cat videos because the the algorithm of social network was very good at inducing this behavior in us while keeping our felt experience of free will. So this is where I think collective intelligence starts to play a very important role because this is a mechanism that keeps us honest with ourselves, right?
4:54:02 · It’s this ability for me to trust enough people around me so that when they tell me, hey, I think this person is trying to mislead you. I think this person is is actually like it it’s keeping you like it it it’s keeping you thinking that you are uh a free being. You are autonomous. You are you have agency over yourself. Whatever word you want to use for that. But the truth is they’ve hacked into your brain in a way that makes them uh you know that makes it possible for them to manipulate your felt experience.
4:54:34 · So that to me is uh one of the question we have to ask ourselves about here. And so that that is in a way why I’m so interested into this uh and and the collective aspect, the normalizing aspect of of the felt experience that we uh that we do for each other every day. I would argue there’s some Oh, sorry. Go ahead.
4:54:59 · Yeah, go ahead. Um okay. Uh so um maybe this is a you problem in in a sense that uh there is this idea of epistemology which says we have to examine how we construct knowledge when we are in the sciences or when we are doing philosophy.
4:55:16 · Yeah.
4:55:16 · And uh feeling is doesn’t rank very high there. There’s a reason for this and this is because the feeling is ultimately a sense. It’s something that is uh an operation of your nervous system of your organism that is translating patterns into classifications without you examining how this works. And as we get older, we tend to have fewer feelings about things because we have conditional models that tell us how to behave in certain circumstances.
4:55:45 · And we do no longer rely on these reflexes that entrance us with an immediately presented reality. This immediate representation of a reality is not truthful. It’s simply the result of a classifier that the organism has produced and that as long as you cannot examine how this classifier is working, you don’t have a claim to say this signifies anything beyond the existence of that classifier. So when you have a certain feeling, you can say yes, I have this feeling and it’s true that I have that feeling.
4:56:15 · But this doesn’t mean that the feeling signifies that something is truly the case in the world. Absolutely. feeling is simply not a way to get access uh whether something is true or not. It’s simply uh an indication that at some level the classifier that your brain is trained is giving you this or that result. And uh so this is it’s a very basic thing you don’t get to knowledge but also by synchronizing our feelings.
4:56:42 · I I I actually totally disagree example of the psychopath that you Yeah, I accept that. But uh they don’t care.
4:56:51 · No, no. I I I actually strongly disagree with that because like what a good but but the question Dan like it’s turtle all the way down. Like what are you exam exam like I agree that it’s a classifier right? A feeling is a classifier but the thing that examines the feeling is also classifier. It’s classifier all the way down.
4:57:09 · If you believe that at the end of it it’s not something like a felt experience that gets you to stop the examination process then I would argue like I don’t know what you’re putting there right it has to be in a way a classifier and when we we think that we can make a difference between the thinking self and the feeling self I would argue this is where we are committing a pro like we are committing a mistake there is no like there is no way for me to examine my feelings unless at some point I stop and I say oh then at that point I’m True.
4:57:40 · And and so I would argue that as collective this is actually what we’re doing is examining each other, right?
4:57:46 · This is we we can be seen like in in an honest interaction between two people where you are aligned enough. You can actually be modeled probably better by you know something like two halves of a brain trying to get to a better examination of something together. So that that’s like at that point I would argue with you that you’re you are inducing this binary difference between your feeling self and your thinking self that I don’t believe exists.
4:58:18 · I think you both have some I mean look this is actually I don’t think it’s getting to the question exactly that you asked.
4:58:25 · However I like answering it’s an interesting conversation. I mean are you would you argue that there are absolutely um no feelings that have any moility of meaning or there there’s nothing that’s actually fundamental process one example
4:58:42 · I might give is empathy I think that empathy there two meanings to this right there is a way in which you can infer the emotions of others based on information about them this is what we call cognitive empathy and then there is perceptual empathy which allows you to experience the feeling feelings that others have and that’s a sense that you can train and sometimes the sense can still be misleading and it takes uh
4:59:08 · years or life to practice the sense and people are differently good at having that sense yet children perhaps because of having mirror neurons exceptionally good no I have been a child I I really was really really bad as a child and reading feelings of others I had feelings about you automatically put a thinking layer on top and what I’m saying arguing for instance no my brain runs at a different frequency than yours and so it is I b
4:59:35 · with very few people basically I have I discovered later in my life that there are some people that I have empathetic resonance with that is very strong but I have only empathetic resonance with a small minority of people and I got better at this as I got older but uh to a very large degree it’s my systematizing ability that allowed me to develop this competence with respect to most human beings and a lot of things that happens with empathetic people is that they assume that other people must be like them.
5:00:02 · They have a very good sense of who they are and they’re often able to pick up on very subtle cues that others are signaling. But I also discovered that a lot of people who claim to be very empathetic failed completely with respect to understanding my own mental states.
5:00:19 · I’m actually talking about it before you are cognitively very active. So babies, right? probably a time you don’t have very much.
5:00:27 · No, I have two I have two children. One of them is uh insane like myself and I has similar issue with empathy by from starting with a baby while the other one is highly empathetic and is really good at reading and manipulating the mental states of others. And it’s for both of them from a very young age. So I assume that they probably were born with different priors.
5:00:49 · I’m also picking on a micro cue from Yeah. Yeah. Yeah. So I’m kind of like interested in this topic of empathy. Uh actually yeah I I used to study neuro mechanisms of empathy a long time ago be but uh but but I didn’t see the connection between the notion of empathy and attributing consciousness.
5:01:09 · So um yeah so so in a way like know there’s no way for us to know like whether other creatures or AI systems have consciousness but maybe like empathy might be some sort like a cognitive bias that we have to attribute sort of like a mental states in other organisms.
5:01:31 · And what one like interesting like study we did was uh uh about the tendency to uh anthropomorphize like things like like the cloud or rocks and things like that. But but those some people have like higher tendency to know attribute such tendency for anthropomorphism.
5:01:57 · And what we found was like know that was also you know that kind of tendency was related to brain regions related to perspective taking and empathy. So I I thought that was like a like really you know like interesting connection and also like whenever we have some debate about AI consciousness or whether some simple creatures have consciousness I feel like there’s a like huge diversity in people’s opinion but maybe that reflects like individuals like psychological trait rather than you know
5:02:30 · their scientific reasoning and and I think that’s what we referred to in your session Ammer this morning when we uh we talked about how to implement layers of interfaces for empathy and care and those those are hard. I think we’re facing the same problems that we’re discussing right now. And uh yeah, gosh. Yeah, we have plenty of examples in the animal kingdom. Um very very I I’ll let this go. I I did want to go back to your question.
5:03:01 · Let’s definitely Can we not do that?
5:03:03 · So So we only have two minutes left on this session and Trisha has been waiting all the time very politely for for asking a question.
5:03:11 · So question for you to we’ll continue.
5:03:13 · we have time for the for more discussion here but I want to to be mindful of people online connecting. So uh yeah maybe Trisha we’d love to hear that question and uh yeah and and go for a final round.
5:03:25 · This is a question for Divia and also Riyota but also anyone else who wants to jump in but you we’ve been talking a lot about biology and neuroscience. I’m just curious from the panelists in particular Deviian Riyota first is just that how does the thinking about your work shift on these topics when you move between um a western medical model that treats the body as separate organs um and we you
5:03:46 · know our departments are arranged that way right um for our medical practice to um a more holistic framework such as eastern or indigenous traditions that treat the body as a whole that’s interconnected and it’s too bad that Michael wasn’t here because I wanted to ask them about you know like that acupuncturist and aryurvedic you know that those practices have have long communicated with each organ individually you know that you can feel what a liver is saying through taking your pulse so I’m curious in your world how does that change when you move between not that if one is right or
5:04:17 · wrong but I’m just curious like how does that shift the way you think about your work as you move from one frame to another where it’s especially the eastern way where it’s more about being connected your whole body is interconnected potentially to yourself and to others the universe to the Earth, the galaxy, everything.
5:04:33 · Uh yeah. So yeah, that’s a like really interesting question. So I definitely feel uh a huge uh culture difference. So um so so in a way the question of well okay so I I think the interest in consciousness is universal. So I I think know where you’re from like people are interested in consciousness of mind.
5:04:53 · But uh but the question of the hard problem or you know like this kind of like an analytical philosophy uh approach to consciousness is very sort of distant uh from my cultural background. uh I I think that it’s a result of like western modernization and
5:05:16 · so so in a way uh okay so for example like integrated information theory is hated by a lot of people but somehow it’s very popular in Japan but but I think the reason is uh here in Japan like people are more open to the possibility of pansychism whereas in the west uh I noticed like you know people just know think IT is wrong or pansyism is wrong. But but that’s but I think that really reflects our cultural difference.
5:05:46 · So like you know like in Japan like you know it’s very natural to think that um like everyone like all kind of creatures like insects and everything has some sort like a I don’t know life or soul kind of thing but uh but but but maybe like you know in the west or in like Christian uh tradition like there’s a know big distinction between humans and other animals. So so I think that makes a know big culture difference.
5:06:18 · I also um I mean I grew up in the west but um I’m it’s sort of an unfair question because I also grew steeped in Indian traditions right so for me there’s both of almost everybody in our family vertically and laterally is a scientist a doctor and engineer there doesn’t seem to be a lot else and all of them believe in Ayurveda simultaneously all of them I mean even the physicists in our family, there’s just seems to be no real separation.
5:06:47 · And I’ve never felt one talking to somebody who just came back from a meditation retreat for 3 days without speaking. And I um when I am in these induced altered states of consciousness um I feel every system all I have to do is put some attention sha into a particular system and that entire system expands into a universe.
5:07:14 · I don’t know how else to describe the feeling, the vibration, the feeling that I could fall into another space that has nothing to do with my mind and um and and it it you know what the you know Vera was talking about panychism that’s and he was also talking about IIT theory all of that was actually it has been borne out by our measurement work to some extent right when I was saying that the things that more and less conscious nervous systems have in common is the ability to integrate information.
5:07:44 · It’s it’s their complexity or entropy and it’s their functional connectivity.
5:07:50 · These things actually tie in very beautifully with eastern systems that include things like pansychism and this idea that we are binding information potentially available in the universe.
5:08:02 · It’s really very it’s very sort of connected to that understanding. Um it’s funny when I talk to either my patients or when I talk to other scientists and you bring this perspective in no matter how much they are steeped into in materialism there does seem to be even in the west some
5:08:20 · resonance I I don’t feel that everybody is as dised from this um yeah one one more thought from Philip and Yosha maybe before we close yeah I mean to me this has been a fascinating conversation you were talking about uh the different nature riota, you know, pansychism, all of this.
5:08:42 · Um I like from my perspective, you know, very western, very, you know, scientist, you know, thinking all the time, very much like like how you describe yourself, uh Josh, even though it might not uh be like that today. Um I I you know to me this this relational approach to phenomenology and eventually consciousness has been a mechanism that has allowed me to reunite this right.
5:09:08 · I no longer believe that consciousness is you know like I or pansicism on a universal scale where you can put every system but I do believe there is some some sort of of spreading of of the feeling of consciousness uh from system to system right and this can very much go across the kind of boundaries you were talking about when you were talking about you know feeling consciousness for any kind of systems or or for um for
5:09:40 · insects or for for a different kind of like living beings like I you know this is compatible with how I I see the world today and it’s also compatible with a very western view which is essentially saying that the only thing we do is we model the world around us and at some point in time we’re like hey the best model for the world around us is to consider this other system as a copy of myself plus some difference and I
5:10:07 · believe this is this modeling step which which creates the felt a shared felt experience. Soon as you model another system around you as yourself. Well then when the system gives you a phenomenal report when it tells you how it feels you feel this as an echo. It’s just like a it’s it’s it’s an artifact of the compression you’re doing inside of you.
5:10:29 · And so for me, this is how I’ve reconciled my very western view uh of consciousness with the more holistic view that can be seen in uh first nation cosmologies or or uh eastern ontologies.
5:10:43 · And I I think this is where we’re headed actually. I think you know maybe through LLMs or through these systems that are kind of forcing us to experience um dayto-day uh the kind of interaction uh a kind of interaction with another kind of system that triggers these same um these same felt experiences. I think we’ll eventually have to reach some point where we consider consciousness as something not very special, not very unique to us humans or to even biological systems.
5:11:19 · I think of consciousness very much as the ability to dream and right it’s the ability to be entranced with a reality and experience that reality is real. And this doesn’t mean that it has to be real. I think that uh western psychology has started to neglect introspection which it did at its own peril uh made it much harder to make sense of minds and consciousness and so on. But uh the I think that ultimately this is not about an eastern or a western view. It’s about things that are true or false.
5:11:49 · So for instance whether panakism is correct or not does not depend on whether I have a western or eastern view. It’s an open question that ultimately has to be decided not with respect to the mental states that I can induce with meditation or psychedelics but that I have to resolve rationally and my rational intellect has to take my introspection into account and your introspection and so on.
5:12:13 · But ultimately whether something is true is not the result of you having a certain dream and then uh interpreting this dream as your direct revelation of the true nature of reality. So when when you for instance meditate uh and you say this has nothing to do with my mind what I experience here that’s a very bold statement because what you’re describing is a particular osianic dream state that is in contrast with your less oianic dream state when you’re not this in this state
5:12:42 · right this of course the distinction between self and outer reality is created as a model inside of your own mind it’s not a state in physics that you’re describing it’s not some kind of access to the way in which quantum mechanics has organized uh in in the physical universe instead
5:12:58 · it’s what you are learning is to manipulate the way in which your mind constructs reality and this might integrate things that you’re normally not aware of for instance the way in which you have feedback loops into other minds and maybe there is a biological internet maybe pensism is even true at
5:13:15 · some physical quantum mechanical level and so on but we don’t know at this point right it’s not like the physicists are convinced of this or that we have a experimental proof And so to me it’s a very open scientific question that we cannot deal with in a religious fashion or in a cutlike fashion or entrancing ourselves.
5:13:34 · I think we have to hold this scientific seriously that because you you introduce layers of loss of objectivism and actually investigating these things as potential truths. uh when you use words like layers like religion and cult which I think potential I had myself with religion but what I
5:13:54 · would say is this um first of all you talk about models I started having experiences like this before I had a model and I didn’t understand what that model would be so that’s that’s piece number one piece number two is um you’re making a lot of assumptions also about the fact that everything has to be perceived by a neural network in order for there to be a kind of perception. The kind of work that that Mike has done would actually indicate that that is not does not have to be the case.
5:14:25 · I do agree with you that it would be very excellent to be able to provide more data around these things. I I 100% agree that you know a truth that is just maybe a shared dream. You would probably call it a shared hallucination. I would tell you every perception nervous system generates is a hallucination. It has actually very little to do with the structure of physical reality. Your nervous system is entirely basically creating itself mathematics about
5:15:00 · that’s just that there are there are I agree with you ways that these things could be investigated and proved but we are unlikely to formalize investigations of things that we have never seen a single empirical example of.
5:15:18 · So if you’ve not had an experience of something you as a scientist are very unlikely to go after and create a hypothesis and find a way to see if there is data that proves that hypothetis whereas people who have had and I will tell you like for instance we have a case report in a human and a patient we now look for physiological phenomena that because it happened and if you’ve never seen that it wouldn’t even occur to you to actually do the experiment so I don’t have a disagreement at this
5:15:47 · The question is where we assign certainty and how do we basically construct knowledge at which point do we claim that we know something. So I’m very much open.
5:16:01 · Have any of you heard of Kristoff Po?
5:16:03 · He’s the one who came up with IIT the internet.
5:16:05 · I think we all friends basically said during conversation that he he had been very much in the materialism domain and he suddenly at a brain mind event last year said um I had an experience that completely changed me and the only way I can describe it is nosis. And he said it had you created such a fundamental shift in my scientific thing.
5:16:29 · He was a complete materialist and I mean I couldn’t there was a limit to like the conversation one could have with Kristoff and then it wouldn’t go any further. And then he said suddenly I’m a different it’s different. I had a gnostic experience and he says that shifted the way he even has conversations and he certainly dreamed that not that he had a superstitious belief that some
5:16:53 · neuroy would produce consciousness but now it’s an invisible ubitious field that produces consciousness but this is not a way to do science his way to but but the way in which you do science yeah but the way you need other humans to do science and you need a definition of humans in order to have other humans, right? That my my entire point is you need to reach a normative alignment as to what you consider another system from which you will accept the reports. And that’s what we’re talking about here.
5:17:23 · Like you have to make a leap of faith at some point. What is behind that leap of faith is what we’re asking.
5:17:31 · I think we have a fundamental disagreement about uh epistemology here. But I don’t think that we will disentangle empiricism, rationalism and history in in within two minutes. No, we’ll solve it. We’ll solve it now. So, so, so we are way over time. Yes. I want to give a bit of time for and then I a comment quick comment from uh from Ammer after that.
5:17:56 · Oh, so we’re closing. Um, yeah. So, like yeah, we we talked like a lot of things. So actually like I don’t have like much to add at this point. Yeah. So uh let me think if there’s anything I want to say. Oh okay. So there’s one one thing I want to say um which is about uh computational functionalism.
5:18:18 · So um you know well it’s it’s a know basically a concept that you know if you implement certain uh type of functions uh you know you may um have consciousness in the system but um yeah but but but then there’s often this criticism that you know you might need some sort like external observer to interpret functions.
5:18:44 · So but but now I I feel like you know we may need to sort of like create like new kind of computational functionalism which is uh to look into some sort like intrinsic uh property of the system. I know it may it may sound a little bit like I but the idea is something like this.
5:19:06 · So like know let’s say you have a triangle it’s a mathematical object but and then independent of how you define the coordinate system uh you know you still have the triangle but but but then you know to characterize the mathematical object you need to have some sort of coordinate to measure the like a shape of the triangle but um yeah so so in a way there’s something independent of how you measure or how you set up your coordinate system but um yeah but but my
5:19:40 · point is that is some something that’s intrinsic to triangle so something like that might be happening to all kind of like AI or biological systems so like know of course you know if we keep interpreting know some neurons are representing this or that from an external viewpoint it’s always interpretative but but I feel like there might be some sort of intrinsic structure to talk about and I
5:20:07 · think that kind of perspective might be you know quite useful when we think about I know the you know potential presence of sentience or consciousness in AI and other creatures or maybe in society.
5:20:24 · Thanks a lot and and yeah I wish uh Mike were here to to continue on that because he addresses exactly that recently discussion last comment from from Ammer before we close. Yeah. No, thank you. This this been fascinating dialogue. So, thank you for joining. Uh I think uh anchoring on Yasha your point about introspection and the ability to dream, right?
5:20:44 · So uh we are in the Silicon Valley San Francisco and one of the probably the most influent voices in AI uh recently said that he engages in zero in introspection and and believes that uh in introspection essentially leads to guilt paralysis and reluctance to take uh bold future oriented risks. And uh he expressed the views that active forward moving focus is essential to productivity.
5:21:15 · And the example he gave is that he basically completely dismisses self-examination as a modern Freudian fad uh that kind of got seeped into in the 1910s and the 20s uh and then argues that modern uh uh figures especially entrepreneurs like Sam Walton and all uh they focus entirely on outward creation rather than uh analyzing their own minds and by focusing on productivity they were able to make bigger impact. act versus uh introspection.
5:21:45 · I’ll leave it at that because that would be a good conversation later but just wanted to kind of build on that because there’s paradigm that’s evolved uh over the last decade or especially in technology and science especi so much science but definitely on the technology and most definitely in in the AI space uh uh around that and we seeing issues at societal level that that kind of culminates from that. So I I’ll I’ll pause on that and uh back to you.
5:22:15 · No, it’s it’s uh it’s uh this this shows me and and and we we haven’t resolved things but we have pointed at uh the difficulty.
5:22:23 · Yeah.
5:22:23 · Uh in the experential uh layers that we need to connect with. Yeah. So so ACI is by no mean sort of engineering thing platform that you build. It’s it’s like Facebook or it’s like an Instagram of No, no, no. Uh there’s there is there there are hard questions to to to to try
5:22:44 · and and really try hard to resolve because the experiences of real minds are at play here and and they’re diverse ones and this is this is extremely important not to to leave leave uh anyone behind with with this when we design this technology.
5:23:01 · So I find this very important. I’ll pass it back to Trisha with many thanks to uh everyone here. Thank you for uh thank you for this this excellent conversation. Uh and back to you Trisha.
5:23:13 · Well Olaf, I would say that it wasn’t just an excellent conversation. It ended with a you know a debate an emerging debate that I think one that really is core to the cognacy agenda right of ACI um which is not just um I think a pistmological disagreement as Joshua said but it’s also an ontological agreement of how we even categorize intelligence and cognition and um experiences as um phipe was saying.
5:23:38 · So with that uh heated debate, I want to take us into you know either debate more in person um which is very important or um we will take a dinner break and uh attend to our bodies and steward you know whatever needs to be done with our families or our homes and we will come back in about 2 hours and 12 minutes.
5:24:03 · That will be 900 p.m. Eastern Standard Time. And then we will move into the next part of the day which is to really bring in more of the researchers that Olaf has been connected with to Japan.
5:24:15 · Um it’s great to have Ya here. Um but there’s even more. So um we are going to you know I I look forward to hearing um the updates from the embody debate as Felipe is saying for those in um San Francisco at the um at the California Institute um or consciousness I think I forgot the exact words um but CI what’s it Olaf machine conscious c oh I didn’t hear what you say oh cim yeah cimc um so anyways please
5:24:49 · have a great meal or debate or a breather and we’ll see you back here in about 2 hours and 10 minutes.
5:24:58 · Hi everybody. Welcome back from our three-hour break and I would love to say hello Japan. Uh most of you are probably are watching right now are based in Japan and you are awake and it’s your morning at 10:00 a.m. So the folks on the East Coast, if you’re like me, congratulations. We may be in our pajamas, but this is the last couple hours of this marathon. And thank you if you have been with me since 1000 a.m.
5:25:24 · this morning with with the whole team um and the West Coast. I hope you are feeling wellfed from your dinner. So this is a friends and family version of Super Intelligence for Humanity. It’s a summit convened by Cognacy, a a public benefit corporation whose charter advocates for humanity aligned super intelligence. And I’m Trisha Wong, your MC. uh drop your questions in the YouTube chat as we go through the next few hours and we’ll make sure that uh your questions get picked up.
5:25:50 · So for those of you who are just joining us uh I think it’s important just to give you a quick you know recap of what has happened throughout these last um you know almost like eight hours you know what have we been talking about you know the hosts have gathered us we have we have Olaf and Emmer who’ve gathered us their favorite friends and thinkers in the space really to figure out how to get humanity from where we are right now with AI which is this uh very dominant
5:26:19 · idea what’s in Silicon Valley which is like bigger models, more compute, more data. And that is one way of doing AI and that is the most pervasive or mainstream way of AI. But there’s a whole other world of AI that is being developed and that’s called artificial collective intelligence. And that’s a type of AI that builds dignity and sovereignty into the architecture.
5:26:40 · It’s what UA was talking about earlier this morning. and not just like trying to figure out expos not just more alignment training or trying to think about ethics you know it’s really trying to say how do we design a whole new system a whole new layer and so I would ask people to mute their mic if you are not speaking so with panel 1A the one uh what we
5:27:05 · talked about with bla1 and pier is that the first conversation really reframed scale itself where intelligence uh was never individual is what blaze was telling us and for Peter he was like look really intelligence comes out of curiosity it’s curiositydriven and then the next panel um was really
5:27:24 · about the second conversation looked at if intelligence is collective and curious then how do we keep it humane and the answer was we have to preserve agency at the edges we have to refuse extraction in the interaction and we have to really let play and embodiment really come through not just through textual uh language And then we moved into panel the next panel uh which was talking about how language is doing more hidden work than we thought and we uh talked about examples.
5:27:55 · I was on this panel so I remember it vividly where we talked about LLM’s you know route reasoning through various languages um through English and how grammar forces these kind of commitments that aren’t even in the original meaning. And then we even brought up how memes uh have a whole tacet knowledge and you know life of their own.
5:28:13 · And so we also got to bring up that if we are to build out this artificial collective intelligence, we’re really going to have to figure out the verification uh problem because right now if we can’t figure out how to get people to trust uh sharing their data, sharing their knowledge, their wisdom, then we’re not going to be anywhere near building out this new kind of ACI. And then the last one, the panel that really precedes this uh that just happened right before the break was a lively one. It was about an almost like I think it was like almost two hours.
5:28:43 · It just like never stopped, but Olaf was um the the moderator and honestly I don’t think any of us wanted to stop because it got into this incredible debate at the end about what even counts as knowledge and it really got to the heart of it. Um but a few highlights where Michael Leven was talking about um how do you how do you communicate with the liver you know how do we start to look at like each organ um so he doesn’t mean metaphorically he he literally means how do we build a chat interface for various organs um we talked about mind uploading
5:29:15 · from Riyota Kana who said hey it’s a gradual process and he laid it out um and then Tivia talked about how consciousness might not be something that the brain generates but it might be something they filter so I think if there’s Um, if anyone is a Pantheon fan, um, this is probably the panel that Yay.
5:29:34 · I see him raising your hand. I think I’m the one who told you to watch this, right? Uh, but I I have I would put my money on it that the authors and the producers of Pantheon who talk about uploaded intelligence, they probably studied Riota’s work, Divia’s work, and Michael and Pier’s work because this is the one panel that cited every concept and also every technology from Pantheon.
5:29:56 · But the through line really from the last two two panels is that you know the intelligence is richer and stranger than we’re currently building for. The current models capture like a little tiny bit of what even counts as intelligence. Um and so much more it lives in the liver. It lives between us and in people and beyond uh just you know individuals. But even that is really complete.
5:30:19 · So this panel asks you know what does it mean to preserve knowledge that has been transmitted through embodied apprenticeship for generations and all day the conversations have been mostly about well you know can we build this like what do we need to do but um the question here is now can we build learning um that’s you know if we’re not if we’re going to build learning that’s continuous and not frozen then the can we build language systems that don’t collapse into monoculture can we build
5:30:47 · consciousness uh measurement identity and verification So, this panel actually talks about in, you know, should we and for whom. So, I’m going to now like switch um as Bruce Schneider was saying like I need a different costume for when I switch into moderator mode. So, I’m going to put on my new moderator hat and now I’m I’m no longer your MC. I’m your moderator. So, I’m Trisha Wong and welcome to uh this panel uh that Olaf asked me to join. And I think Olaf knew and knows about my research in China on tacet knowledge.
5:31:14 · I think in one of our first meetings, I told them that I chose uh my research to do in Japan on chicken sexers because chicken sexers have an incredible these are sexers who have to identify a male chicken versus a female chicken. And if you’ve ever lived on a farm like me with some of the best chicken sexers in Japan who go around the world training um if you have an egg, you know, you have to think Japanese chicken sexers. But it is such a a craft and it requires such tacet knowledge.
5:31:44 · It’s so embodied that um the Japanese are really the best at it. So this is one of the reasons why I’m really excited for this panel is that you know what makes Japan one of the hardest test cases for everything we’ve talked about today is that ACI is built around tasset knowledge. It’s around govern attribution, local epistemic authority, compounding memory, all these things that you know is known that Japanese is known for which is the tacet knowledge which is why I you know did my one one of my last research projects was on chicken sectors in Japan but there’s
5:32:16 · so many more examples that’s like just one of many examples of how Japan is this incredible case study for tacet knowledge knowledge that survived uh over over decades over hundreds of years and I brought up memes earlier about how memes are a form of cultural transmission. Well, one way to think about craft in Japan, it’s also a form of cultural transmission that has lasted and regenerated itself and sustained itself.
5:32:42 · So the question of the hour today for this panel is when we say we want to build ACI that preserves tacet knowledge with consent and revocability and with agency does Japan show us how to do that or does Japan show us why it can’t be done or does Japan actually give us some you know have like the answers for like this is the way we must do it or else.
5:33:08 · So to get that info and to get that um to get some agreement, I have three people um who are so great to talk about this and I’m very excited for our first panelist, Hiyaki Katano, who is I think one of the most, you know, consequential AI scientists in Japan today. He’s the chief technology fellow at Sony.
5:33:29 · He’s also the president of Sony Computer Science Laboratories and the CEO of Sony AI and the president of Systems Biology Institute in Tokyo. and he’s created a IBO, which is a Sony robotic dog that’s taught, you know, generations what it’s like to live with this non-human intelligence. If you haven’t played with one, you definitely should. And he also found a Robo Cup with this audacious goal of building a robot uh soccer team. And we’ll ask some questions about that later.
5:33:56 · But uh most importantly, his recent work is launching the Nobel Touring Challenge, which asks, you know, whether AI can do science that’s worthy of a Nobel Prize.
5:34:07 · And so with that um I’m going to I’m going to first introduce actually Ammer and Olaf and then we’ll I’m going to start I’m going to throw a question over to you Haki. So Ammer who is also one of the co-founders. Ammer and Olaf are both co-founders of Connelly and you’ve heard from Ammer already potentially this morning if you’ve been with us. Um but he’s one of the people who’s been deeply thinking about applying all this kind of frontier AI work into actual real business settings. And so he’s doing it again.
5:34:34 · He’s done it before at various companies that you all know of. um you know Nike to name one of them but now he’s doing it in terms of cognacy and applying and really taking this whole field beyond the little um bit that we know about AI which is just you know about large models and large compute and he’s asking how do we do this for ACI and then we have Olaf who is another heavyweight in Japan one of my favorite um AI scientists you know in the world who I’ve been whose work I’ve been following for so long um and so Olaf is also he’s he’s a co-founder founder.
5:35:07 · He’s a founder of cross labs in Kyoto and he’s also president of an international society for artificial life where where his work is focused on collective intelligence and he’s been working alongside you know a mainstay of the Japanese research ecosystem for years including many collaborations with various Japanese AI scientists and he’s one of the key people who can really speak to Japan not as just a place to build for but as a place that is already already building a lot of what we’re talking about today.
5:35:37 · And so with this, um, I want to throw my first question to Hiyaki. Um, I’m just going to get straight to it, which is that you have this most recent project, the Nobel Turing Challenge. Um, can you tell us about about what made you want to create this project uh to ask, you know, whether AI can do science worthy of a Nobel Prize? And tell us, you know, what what is this um where do you think we’re at, you know, since you started it and what what what do you anticipate is going to happen?
5:36:10 · And since you are muted, but before you unmute yourself, I’m going to encourage all my panelists to be conversational. Oh, okay. Good. I want to encourage everyone to be conversational, interrupt each other. It’s all friends and family. So, um, don’t wait for me to ask to ask a question or call on you if you want to talk. All right, Hidaki, you’re on.
5:36:28 · Thank you. Thank you. Thank you very much, Trisha. And the uh Nova children challenge is like one of the grand challenge I’m actually proposed like a 10 years ago. And the the goal is to uh you know asking the questions like built the AI and robotic system which can autonomously you know do the research by itself and come up with like a series of major discoveries some of them worthy Nobel prize or beyond you know the uh this is
5:36:55 · actually the uh you know interesting title like this is like a name challenge in Nobel and chilling is like you know I initial studed like some grand challenge uh for asking AI to build you you know asking like can we actually build AI to be able to win a Nobel prize.
5:37:10 · I gave a talk at the Stockholm actually then you know front row uh you know there’s like you know people from the uh Kalinska Institute who’s actually a part of the Nobel committee and then they come up and say the kitanosam I mean you know uh Nobel’s deed actually if you read carefully is given to the human being not to the AI and then okay then uh we have to uh uh challenge if uh
5:37:33 · you know comedian will be able to tell the difference of the AI making discovery or human making discovery you know so that would be more like a chilling test right so uh you know I just didn’t decided to name this challenge a Nobel Nobel cheing challenge is like it’s a challenge and the question the challenge is that can we actually build a machine uh which can
5:37:53 · carry out like a high performance research some of them worthy Nobel Prize or beyond uh fully autonomously or highly autonomously and then the question is if you have to build this machine like this the machine behave like a real human scientist or is it going to be a very different kind of intelligence which you can actually tell yeah this is machines not human but still making a series of major discoveries. So uh no ching challenge actually is a challenge and question.
5:38:22 · Okay. Now you know like I have proposed like over years ago the roboc which is like a robot actually being like beating a world cup champion in Saka uh which actually in the US we’re going to have like you know world cup soon. So I think this is very interesting to observe how human behave. uh but like that is the intelligence in the uh uh uh that
5:38:45 · requires like a bounded like you know embodied physical intelligence in the teamwork and the real time and act in a millisecond order and everything finish in a 90 minutes plus right so this is like a times horizon is about like that that is a time horizon we are talking about now and then also like I because it’s a defined as a game of soccer it’s a context to be bounded okay so the other challenge which is no chilling challenge is completely unbounded.
5:39:10 · This is completely open space and some of the decision have to be made very quickly like you know if you going the chemical or anabolical reaction you have to really intervene to the reaction to understand what’s going on but at the same time it can take like a month or even a years to make a major discovery and it’s completable. So it’s two contrasting really the grand challenge which actually shed lights on a very different aspect of intelligence. So like a Nobel Turing challenge uh is basically asking that we build a W drive for the scientific discovery.
5:39:43 · I you know if you uh some of you might look at the uh uh you know watch like my uh TED talk uh TE AI talk uh end of uh September last year and then uh I actually talking about like all the web drive like a you know you know trillions of datas and then you know from that data extracted knowledge and then come up with like a you know you know billions of hypothesis and then uh millions of experiments and a lot of like a thousands of discoveries.
5:40:13 · So you know the point is we are may not be good at science uh you know we are talking about like a scientific discovery if for the biology environmental science material science is vastly complicated vastly complex like you know it’s a number of the molecule involved like you know all the you know state space is like a massive and it’s open-ended and our cognunive capability is very much limited like for example biome area we have like over two million paper published per And we can’t read them all.
5:40:44 · And then uh you know when we actually do the research we come up with hypothesis right that that need to verify and usually we come up with one or two hypothesis probably three max but it doesn’t really populate like entire possible hypothesis space and then if you know we’re going to go to experiment and verify that and if it doesn’t work and we go on right or like you know someone else coming in like we have to restart like that’s why like a biomedical research takes like you know decades actually not years you know.
5:41:09 · So like sometimes like you take like you know 20 20 30 years to actually come up with a new you know major discoveries right. So like I want to just like you know change that like instead of come up like a one or two hypothesis we come up with like you know hundreds of hypothesis that populates entire hypo space and then trail brazing you know that space and then run experiments in parallel like a thousands of experiments to verify them all. So like we can even bend to the hypothesis space. It’s it’s a that’s why I name it the web drive.
5:41:40 · Okay. So like a building driver of science is actually will completely change the way we do science.
5:41:47 · So do you think what’s the role of the human being in if any in setting the hypothesis or giving any of it direction or are you imagining this prize is completely devoid like at some point you’re like this is not human engaged at all.
5:42:02 · Yeah.
5:42:02 · So I think it’s interesting human may be able to direct like this is area we want to explore right like we can you know we can set the directions like we want to solve the problem we want to solve like a curing the cancer or like you know in terms of like efficiency because like even AI robotics it’s not like a uh you know some other resource bounded right so you want to actually uh set up like a theme for example like you know let’s uh find a cure for the specific type of cancer or cancer in general that actually means like a much bigger of hypo space or so like that we can actually say like what to be solved.
5:42:36 · Uh the other thing is like AI can actually free right you know you you know AI we let AI to come up with a problem that worth solving and then let them solve like just moving on so it’s completely free so like AI may be able to find some problem we have this still important able to solve it so like the
5:42:57 · issue for the human doing science is that we actually align too much with our value system which is potentially is a bias or like you Yeah, you know we tend to work on the genes or like a molecule which we believe is important and then I leave out like something which is untouched like you know but that’s untouched terrains like there may be a lot of interesting discoveries like you know in fact like the history of the science proved there’s a huge very disruptive discovery coming from the area that people haven’t touched but like very difficult for us to do it.
5:43:29 · Yeah, but very difficult to touch that unless you have a very specific motivation because we want to be successful. We want to be contributing. Therefore, we tend to choose the a topic which we think is important.
5:43:45 · So, sorry like our AI agent is like stealing this conversation and start talking to me. Okay. I guess I’m populated by like all the agents like trying to intervene my conversation. Um well well you know I’m I’m curious before I turn it over um to Amar but I I have to ask one more question which is um you know what
5:44:05 · already we’re seeing weave mind is a company that I was just talking to where they’re creating an agent and to agent communication it’s entirely agent to agent language and so at what point do you anticipate um you know what responsibility do you ensure that when the machines are on their own autonomously communicating their own way because I agree I think like the way that We even design our systems are too much based on our own point of view. You know the way we code is based on how we see the world. But if we really let the machine go free, it’s going to go in a way where we can’t expect.
5:44:36 · So how do we expect that it’s going to come back and actually tell us what it did and how it’s going to translate like we wants to align ourself to our value system, right? So like that we want to use AI for our benefit, right? But like uh uh I I just like I wonder if we actually let AI to do whatever they want.
5:44:59 · Are we going to have a better you know the society or like AI ecosystem like a wealth of knowledge which is better off like a much better unbiased than uh we intervene like you know we actually have like a pretty much anthrop uh you know centric uh value system that you know we want to actually have a system benefit us like a you know benefit us means like a foo among us you know so you know I
5:45:28 · just wonder this This is a really philosophical question if like let’s do whatever is you know is going to be maybe like a diverge from what we want uh exactly or what if it decides like what if it’s finding is like maybe I shouldn’t share it with them because it’s not going to help you know because it’s done all the calculations and all the scenario modeling.
5:45:46 · Yeah.
5:45:47 · Or maybe it decides it’s not the costbenefit analysis. So yeah decide okay I mean you know let’s not share this humans like they’ll do kind of weird thing with this. I mean you know if that is maybe a reasonable thing like you know we should respect I mean you know because like we tend to actually be obsessed like we try to take advantage of like a discovery for our benefit you know of course like we try to contributing but like you know right it’s you know human being are really tricky like we have like a you know if you go to the kung for example you know
5:46:17 · you know we have a persona which is very public face but we have shadow you know and then you know like it’s complicated so uh uh Exactly. Because it may it may decide like, hey, I know historically when there’s been like pharmaceutical findings, it doesn’t actually benefit all of humanity. So if I tell it to these people, it might only go to the the people who the company who has the most power. And so we’ve seen that and AI may catch us in that, you know, the the one you design. So it may actually make a decision.
5:46:46 · So we this is too exciting.
5:46:48 · Okay.
5:46:49 · Um well, we’re gonna come back to this.
5:46:51 · I want to I want to ask Amber then, you know, here we are. Um, you know, this kind of might sound like, you know, it’s thinking about as Hiaki is talking about designing his this autonomous machine that’s going to be able to just, you know, make these discoveries and, you know, you’ve been thinking about products very seriously in terms of how how do we productize, how do we make ACI real, right? Um, and how do we how do we use it? Meaning like how do everyday people touch it or companies or institutes touch it?
5:47:17 · And you talk a lot about I mean this is the product that you’ve created although the ideas you talked about is around the cognitive vaults and semantic expert twins to actually figure out how do you preserve this tet knowledge um to ensure that you know the AI doesn’t go off rails. Um and so h how can you talk to us about how do you think about building this in a way that’s not as extractive and something that people have real- time ownership over?
5:47:52 · Fine.
5:47:52 · Yep. Thank you. Uh, can you hear me?
5:47:56 · Second.
5:47:58 · Maybe there’s a lag.
5:48:00 · Yeah. Let me just fix this.
5:48:03 · Maybe it’s fine now.
5:48:04 · Is it okay? Can you hear us?
5:48:06 · Uh, just make sure you speak up.
5:48:09 · Okay. Can you hear me now?
5:48:11 · Yeah. Lean in. Keep leaning in. We like that. Then we get to see more of your face.
5:48:15 · There you go. Uh first thing uh great conversation so far as I started and uh before I get into my response to that I wanted to kind of bring a concept with X-Prise that I designed about five five and a half years ago at this point um was called uh planetary health sentinel and the whole idea behind the planetary health sentinel coming out of the pandemic or because we were still in the pandemic was that how to not be surprised by such novel events.
5:48:46 · And and the concept there is that the black swan events are essentially considered black swans because we as humans don’t know where to seek and understand the patterns that are unknown and unseen. The patterns are out there.
5:48:59 · They’re evolving over time but we don’t know how to ask the right questions because our uh uh recency of our experience is essentially cloud our sense of inquiry that that goes into the longer term thinking. So the whole notion it’s it’s so similar to the concept we just talked about was that can we break build an AI that has localized contextual data from around the world that is rooted in its own basically uh uh uh uh um local uh local
5:49:30 · context everything from a biologics to to zutonics to societal level information where AI is actually doing the initial work in in uncovering patterns and generating hypotheses. Some may be gibberish, some may be interesting.
5:49:47 · Um and then the concept there was very similar to what we talked about where human role in that design was essentially as a as a validator in terms of is this a question that I want to explore more uh through AI and essentially once you say yes explore more then you push it back into the AI to where then AI can start to not only further distill that question but also through simulations and again collective learning in a way start to actually create hypotheses around potential pathways to solutions as well.
5:50:18 · So in this case the idea was essentially to go towards this you know the unknown unknown where the questions are unknown and then so are the answers because right now the limitation between human AI interaction is essentially the human because we are the ones who are essentially the limiting factors in terms of what questions or what inquiries we can come up with versus opening it up to the AI to essentially come up with the inquiries and the questions.
5:50:44 · So, so as far as the the pushing the boundaries of intelligence and you know I think I think uh the question posed essentially around do we have create a system where it’s a system of AI versus a system of humans and then there’s an aspect of it which is a governed coordination layer between systems of AI and systems of humans where where this this collective systems kind of evolve out of it.
5:51:10 · So you know some of those are the part of the inquiry I think that that we we are uh discussing and architecting and as far as the artificial collective intelligence is concerned I think the way I’ve been thinking about is is more in terms of learning from that design principles right and I think it’s it’s almost like an AI scientist in that case and it was a the AI scientists were basically uh uh epidemiology researchers right and people who research viruses and and and
5:51:37 · their spread and public health so the idea because multiple expertise coming together into a collective ecosystem where each ecosystem or each expert is an expert AI that is coming with the question and the answer right so so ACI essentially kind of builds off of that that that work that I I was putting together for X-prise a few years back and the design as we’re thinking about is similar to how uh almost taking like a socratic approach of of uh inquiry first before rushing into solution Can we do we understand the question itself?
5:52:12 · So if we root and ground systems of intelligence in their localized alignment, grounding and context and create and enable pro-social uh socratic basically behaviors where the rewards to the intelligence is based on collaboration and cooperation and and
5:52:32 · essentially an epistemic right approach to to to debate uh into factuals and counterfactuals and perspective shaping and perspective switching. uh the hypothesis is that through that approach and again we had a longer debate also uh tapping into evolutionary biology around uh around what we learn from the nature is how evolution happens right through distributed learning.
5:52:54 · So within that construct is the the AI agent should be able to uh cooperate, coordinate perhaps create new agents and and almost like a parent create a child that can continue the momentum forward where where where where the the parent kind of like dissipate away uh versus agents that could potentially even reincarnate later on in the stages uh when when that particular aspect that they were uh they were dormant will kind of come back.
5:53:21 · So we’re thinking about like those types of architectures in terms of how this collective intelligence essentially can work. uh and in within that the thing uh Trisha you asked about the the question around sovereignty and provenence and collection I think which is coming back to the cultural root of it is the design that as we talking about the core design principle is sovereignty of uh intelligence knowledge and wisdom and
5:53:48 · and and so much of this abstract knowledge and embodied knowledge remains with cultures and communities that are still rooted deeply rooted in earth space principles right? Uh uh artisans and the crafts and and singing and dancing and and and in all of the creative expressions and those creative expression I know earlier panel we Erica talked about uh you know we forgot uh
5:54:14 · the aspect of of play right the and critical role play uh plays in in learning so from that context I think the idea is around is not move away from this this this paradigm of extraction versus enabling a systems where it’s a collective system where people have sovereignty over their their judgment and then their their their action so that we can we can not only learn from them but share with them right for them.
5:54:44 · um and then revocation and everything kind of comes into play. But it’s the idea is instead of taking an extractive one-way monolithic system can we create a distributed owned and sovereign systems where uh as as the information that gets collected versus how it gets used versus how it gets implemented and utilized and how do we make sure that the ecosystem itself benefits from it which makes it a essentially a thriving ecosystem and flourishing ecosystem versus a uh oneway extractive uh ecosystem.
5:55:14 · So, so anyway, going to longwinded here, but uh just just yeah, how do you imagine I’m curious then in terms of what’s your what’s your thinking around how these systems, you know, the idea is that they’re built on hyper local knowledge? Um will the AI system how much of it is calibrated to prioritize the hyper local that communities needs? Yeah.
5:55:38 · Um and ways versus also balancing for the larger society and humane humanity, right? talk about these two different scales and we’re constantly jumping especially especially with ACI. You know what I’ve noticed today is that we’re talking about two different temporalities and scales.
5:55:54 · Um and sometimes these things work together and sometimes they don’t.
5:55:59 · You know sometimes the hyper local when you’re in a scarcity mindset people will will not take care of the commons.
5:56:06 · Communities will fight out and will like actually harm each other and work against themselves. But then when you see communities working in a commons abundance mindset then the way they steward the land actually uh furthers you know and and contributes to humanity and the overall um goal of the way we steward our earth. So yeah um how how do you how do we anticipate for that because any we don’t we can’t just assume communities are great. It’s like AKK is also a community you know and they’re local and they’re hyper local right.
5:56:37 · Um, so we we need to think about it all. I can’t help it, but as a sociologist in the room, I’m like, this is not good or bad. This is like everybody’s going to be at the table, you know?
5:56:46 · Yeah. I mean, how do we think about that?
5:56:49 · No, that’s a great question. I think just to kind of purely thinking from an epistemic perspective, why not have those perspectives in the system? Like why exclude it? Because they are part of the society. In a in a way if we have to to represent who we are and what our society and what different belief systems are then then they have to essentially in this theater of ideas have to essentially uh uh uh compete and collaborate uh with each other.
5:57:16 · Otherwise if we we put too much sanitization of that I think I think that leads to the the issue of compression essentially right and and and science is a process of you’re not creating a perfect world or or or or over or over or over or over or over or over or over or over or over or over or over or over or over or over or over or over or over or over or over or over or overfitting to a particular political or philosophical belief.
5:57:35 · Right. Right. Um and and I think we we have seen the the impacts of that where uh uh postraining behavior adjustments as a as a bandaged to to uh alignment occurs and then now all basically large language models essentially are leaning to a single uh political ideology.
5:57:54 · Right? So so I I’ll just just put it out there and I think I think uh I’ll respond to that. Why not? Right? Because that that’s the society uh the way we are. I think going back to the question around um localization versus almost like a generalizability at the global impact level. uh the way I’m thinking about is that the the information has to be uh
5:58:19 · essentially grounded and and aligned locally because a a plant for example of Tulsi uh in India has religious and societal uh uh representation.
5:58:31 · uh it also has amazing uh uh medicinal properties and there are certain aspect of taking care of that plant because of the spirituality associated with it plus medicinal applications and usage. So if if those all of those nuances are not anchored locally then then you may have a systems where they may come up from the outside and they may assert their perspective without really respect of the underlying like the local local context.
5:58:58 · So in a way if let’s say in an ACI or collective intelligence ecosystem let’s say in a near future state we are working in a drug discovery problem for a rare condition disease and and it goes through our ACI where let’s assume there are seven or nine different symptoms of a particular disease that have been identified or characteristics and three of them are resolved by a plant based medicine out of India another two are connected to uh medicine out of Japan and then some from uh from Indonesia and
5:59:30 · then a couple some overlap from from some medicinal practices out of of Brazil. They in this case there is a local alignment in terms of context as well as alignment in terms of like the how it’s cultivated, how it’s harvested, how it’s processed, how it’s prepared, right?
5:59:49 · So that’s all very local and within the context of there’s an aspect of like an is spiritual and cultural in this case too like how is this like what type and many of those can be representative of the environment and atmosphere and and in various other forces right but if we don’t we don’t consider that as part of bringing it all together into a all of almost like a all of these different recipes coming together potentially for a new compound that could solve this problem.
6:00:14 · So, so thinking think about from those time of a layer where the underlying is extremely important for that type of a cooperative collective intelligence to evolve at the global scale because if we don’t do that this this process of inquiry and factuals and contrafactuals that evolutionary aspect will not happen. So in a way they are coupled together one cannot occur without the other otherwise it will just become another compression system that we have today.
6:00:41 · Yeah.
6:00:42 · Got it. It’s about that there in your mind in your conceptualization they’re inevitably it’s they’re intertwined that you cannot separate it. It’s not that there would it would be designed in such a way where um it would there would not be a competition between the hyper local
6:01:00 · versus the population kind of demographic humanity level right is that what you’re saying from an application perspective Tisha it could be that let’s say you know going back to the to to Japan right in this context and just visited for the first time recently so don’t have as much context but just just congratulations you’re now allowed on this panel that’s Thank you. It’s the recency that matters in memory.
6:01:24 · Uh the you know and this the same conversation I’m having with my friends in K Korea and and and also in India and various parts of the world where cognitive decline as the aging population is a is a concern right and we understand that cognitive decline can be uh can be slowed down by engaging people into arts and crafts and cultural activities that they grew up with. So doing calligraphy and and dot space paintings or or or or or uh embroideries and and things like that, right?
6:01:56 · So these are these are like localized practices and context that can be utilized through collection and preservation of those practices that have a direct measurable impact on cognit on essentially reducing the the slowing down the cognitive decline.
6:02:12 · But that information that gets collected, we’re collecting hypertext and sensing and tactile sensing and all of these other embodied intelligence with that within the context of that work that in a global sense can be utilized to train next type next era of essentially physical intelligence that would have the level of tactile sensing and dexterity that uh that is very hard to replicate today. So just an example of how it can be rooted together but also coupled.
6:02:41 · Mhm.
6:02:45 · Yeah. So, so with that, I mean talking about Japan, which is um one of our topics is that uh I I want to ask, you know, um go to back to Hiyaki, you were talking, you know, I want to we want to uh I really want to ask you in terms of, you know, if we’re talking about building out this collective intelligence infrastructure and let’s say Japan is the first place, you know, country that’s really going to take this seriously to do this, how do we build this out in a way that preserves craft?
6:03:15 · um the scientific expertise and the also unique like how do we create a unique design knowledge while ensuring that um there’s also discovery like how what do we build first because Japan is this place site that truly values this kind of tacid embodied knowledge and and almost it feels like everything it does right I I I think uh you know in Japan we have a lot of traditions and then a history and then all the deep rooted
6:03:44 · craftsmanship and that’s uh you know I think that’s not something like we can really buy money you know that’s a history and then know the traditions so particularly like a Kyoto and west this is a really the cultural center of Japan has a lot of like you know familyowned craftsmanship like you know you know workshop and all that partly and style
6:04:06 · and all that but it just like pity like you know Japanese uh uh you know industry those industry are considered to be kind of like a winding down kind of phase like it’s like a a little bit like a you know stagnating and uh but if you look at the uh Hosan’s place and the
6:04:23 · Nishiin te style like you know he usually you know he used to actually weave the nishi texture this is a very specific kind of te style which is really seven layer very sophisticated but he actually got in a 32 cm to use like a obi like or the belt for the kimono but then uh he actually noticed like international size for the textile is 150 cm. So he build a weaving machine to be able to weave that 150 cm wide instead of 32. Guess what happened now?
6:04:53 · His text style is on the bugali hotel that leads cton for season wallpaper.
6:04:58 · You know he completely ref how to use it and they get the value like maximize value. He have like a many of those machines now and then the international presence like he was in the Milan salon last week and then you know things like that all the global celebrities come to his Galilee in Kyoto. So like it’s just a matter of like how you frame it.
6:05:16 · If you frame it traditional like uh you know context I mean that’s valuable but at the same time like you know people’s attention is moving away but you know you take like a technology you take like a craftsmanship but completely frame it you know so that you know that’s really a wonderful thing he did and there are many other things craftsmanship in Japan all the traditions and potly and all that that can be reframed and you know
6:05:44 · put the value in tfold and the strength is like Hian traditions there and there’s like you know teroir I mean which is all the locality because all the craftsmanship and the local community supporting it sometimes like you know partly you have like all the local soil which has a very specific chemical and microbial competitions which really required and
6:06:05 · you put the technology in a broader sense technology not just like you know all the high-tech thing it’s just like technology to branding technology for the management technology for the capital in you know injections so if you I I say like a 3T like you know tradition to and technology will be key for the revitalizing things and now now
6:06:25 · sometime like I talked to like government officials that noticing like my uh you know discussion on this line and then I says like how about like if you put like a robotics can we actually increase the value I says no I mean a technology not going to increase the value you know you have to really reframe it you have to increase the value of the products or service you’re providing then you have money to buy
6:06:46 · robot and AI you know robot now is expensive and robot and AI or technology not increase the value of the products you know you have to reclaim it and if you increase the value of the products itself you have money to use AI or robotics to be able to you know improve the quality must produce it or like some added value on top of the uh uh something that you reframed already.
6:07:09 · I mean what the example you just gave is literally what Ammer just talked about which was he was saying you want to have something that’s at a hyper local level well preserved well protected yeah but then connected at this other you know into these larger ecosystems or infrastructure that it wouldn’t have had access to otherwise. So I think in your example it’s like how many more examples could there be of this you know um if we had the right kind of infrastructure around it to reframe it. Yeah.
6:07:37 · Um, so I really love your example and the three T’s we are adding that into the research agenda is you have to reframe it with tradition, tervoir and tech.
6:07:50 · Yeah.
6:07:50 · Together it’s not it’s not just tech by itself like not just tech. I mean and you know the take is like a just a money like you know how much money you can pour into this like all like you know data but like you know you know Japan’s we really need you know uh take advantage of the
6:08:06 · history and uh you know traditional ter and that actually not something you can buy you know that’s a strength we have like a really history and people pay for that story you know so like if like a you know singing products for example like a partly a te style if you must manufacture With that story, they’re going to be cheap. They’re going to commoditize. But if you if there’s a story behind it, story of the human being crafting this or like a history story history of the region, you know, then people faith for that that value will be 100 100 times bigger, you know.
6:08:39 · So I think like a really the thing I mean also like those history is not on the web usually. It’s like a you know just like a fork folk it’s a you know people talk to each other and know people document it something online but it’s not really in a systematically on the web and know and then you really have to talk to people you have to really touch things locally there many you know craftsmanship that’s family owns like all the old man actually doing
6:09:05 · the crafting it and then the family says like okay if my dad actually is like passed away we close this oh my god you know you know that that’s key. The next five years really the key for Japan to preserve those craftsmanship. I think it’s really uh you know important I think so that’s I think the very
6:09:25 · different angle the very different aspects like you know AI is about like having a large scale data so that’s actually uh you know you know something have to be in a cyber have to be in a in a data you know for good reason but like you know the value Japan has like one of
6:09:43 · the value I would say uh is in the era that’s not on a data that’s a physical reality in the locality and then then how to actually make the value like a 100fold bigger will be the key and then you know uh you know then the Kyoto will be epicenter and probably area which is more the west n and is that area it’s
6:10:05 · beautiful area as well is probably one of the best resort area if you do it right and then has a lot of history out there I I think I think every part of Japan would argue that they have the highest of tradition so you’re going to get competition within Japan for that. But so you should be careful with your own people. But um I I have to say you know is you just literally outlined the research agenda in case for why one ACI is urgent.
6:10:30 · Artificial collective intelligence is urgent because so much of these traditions um are have are biologically bounded because these people are going to pass away. We are we are stuck in not stuck but we are in bodies that will expire. And so one you just um outlined the urgency for the artificial collective intelligence research agenda and two you outlined the urgency of it to happen I think as a Japan as an ideal place for the pilots you know for this to happen.
6:11:01 · So thank you for that and then you just gave us a research framing with the three T’s tradition to our tech. So thank you for that. I think we can maybe revitalize like with a new technology like with the hosan actually we decide to go for like a creating new sik worm factory. It’s a sick hub in the northern part of Kyoto.
6:11:22 · We secured a huge chunk of land. We actually put like all the mobility and for the sick woman to eat like so we got like a microbiome research on the soil to make like a you know best like a plant biology out there and we got like Japan has like a biggest like reser all the uh gene bank on the sigworm and in
6:11:41 · agriculture research institute in Japan and then we’re going to have like a entire robotics and sensor system have like a millions of sigworm all the probably most modern and the most scientifically valid silicone factory in northern part of Kyoto That’s a project I’m working with like a hosan right now.
6:11:58 · Wow. Uming okay there’s so many there’s so many examples that you have. Yeah.
6:12:04 · Um this is incredible because that should be a a pace that we do that case study. Um yes Emmer, please.
6:12:11 · Thank you. No, this is amazing and thank you. I think it is so much alignment to to what Olaf and I have been discussing around our research uh thesis. I think the concept of uh human wisdom vault essentially uh is is quite relevant to this discussion and uh the human wisdom vault essentially as we talk about is a analogist to the the Norway seed bank
6:12:35 · which preserves seeds in case of a climate catastrophe so that they the seeds can be utilized to basically regenerate agriculture and forestry and similar to that um you know we we are hearing around systems of knowledge or or intergeneration wisdom uh that is disappearing or at threat of disappearing around the society.
6:12:56 · Everything from 14 languages essentially disappearing every year and then every time a language disappears all the traditions, practices, knowledge essentially of centuries uh goes away with it and and there are traditions I think uh you know the Japan about 5 years there are certain parts of the world about 10 years netnet I think the studies I’m seeing is we have about 10 years of time frame left to preserve generations of human wisdom before it’s gone forever and And once it’s gone,
6:13:27 · uh it will not be available essentially to help us uh solve the complex problems of society going forward. Everything from from from climate and agriculture and food to uh and health to to uh u acts of judgments and then acts of uh of of essentially skills, right?
6:13:45 · All of that is under threat and uh so the concept of human wisdom w that that uh that that that we’re working on is essentially uh to collect and preserve these systems of knowledge uh around the world before they disappear and then utilize um uh multimodal approaches to to AI to essentially to to make sure that these systems of knowledge and wisdoms continue to to to help us solve uh problems of today and tomorrow.
6:14:12 · So it’s it’s it’s a very uh beautiful alignment um uh to to to what Hiaki is saying. Yeah. So thank you.
6:14:22 · Oh what a scary statistic Ammer that is crazy. 10 years to preserve human wisdom. Well if it’s 10 years just prob like that’s the probability on a human um like global level then I wonder what that number is for Japan because it’s much less.
6:14:39 · I think you know like you said about any guesses I mean all the yeah definitely like a five year would be a key like all the months like you know kind of pretty much like uh you know elderly population right now but you know ah so you meant five years for all I thought it was just for that specific
6:14:56 · you know very specifically for this like uh you know all the uh craftsmanship like but at the same time like all the young guys actually coming in because this is the opportunity like people not stupid people really creative you know Yeah, at the same time like you know if you got like all the AI evolving AI I mean you know for in terms like intelligence like you know it’s just matter of time like AI over supersedes like a 95% of intelligent activity for like 90% 95% of people.
6:15:21 · So like uh you know I think like you know uh my bet is like in next 20 years like you know human will you know step down from the driver seat for the you know civilizations. So like you know AI in a biological uh you know robotics or a biotech based entity will probably drive the civilization forward and uh I think like next is a key that we want to build like a uh you know machines uh that can actually bring our civilization forward like a much faster than we do.
6:15:55 · Yes
6:15:55 · I fully fully agree there. I think we’re also seeing that in other tradition like uh some Buddhist temples because they don’t have the uh uh younger generation to pass the traditions to and now they’re training human robots to become basically priests because they want to preserve and carry essentially uh those practices forward. So it’s it’s quite fascinating to see how that change is already beginning uh uh right now.
6:16:21 · Yeah.
6:16:21 · So I mean with that Olaf um you are the convenor of this conference also along with Yammer and we have not even heard from you. I don’t know if you’re just so entertained from the show but also I may have not done a good job of call of of calling upon you because I was mesmerized by the conversation especially I’m sorry but I I’ve been wanting to meet Hiaki for a while.
6:16:39 · So um I apologize for being biased towards him but um so I I want to hear your thoughts on what is the right balance you know um I mean Hyaki is literally saying that he has a bet um that in 20 years humans will step down from the driver’s seat says the founder of the Nobel Turing challenge and then we also have Ammer here talking about uh the work that you’re all going to be building out which is a cognitive vault um to preserve wisdom um and he’s got this 10-year timeline Mike right he decided about preserving human wisdom.
6:17:10 · So where do you see all of this going from your perspective um from not just studying Japan but like every all kinds of you know artificial life in general?
6:17:22 · So there is um so so I’m I’m very happy to to be in the in in the back seat here because yeah first I wanted to hear from Kitanosa and and and Ammer here. Um there is a yeah so so so a lot of kitanos I don’t know if people in the audience are realize how much he has done and what uh what is behind the whole force of Sony what that represents for Japan uh it’s really uh there’s
6:17:48 · there’s a lot of powerful uh I think spirit that could be replicated uh elsewhere but that could originate and be symbolically uh coming from Japan there um you mentioned hos and and all the well the craft tradition uh and there’s a lot of a lot of that I guess the yamochia also yamachi banjo is
6:18:11 · also very active here in the families in Kyoto I guess kansai in general but but all of Japan has a lot of strong examples right uh for uh how preservation can come about um and yeah we yeah I’m I’m very uh uh yeah first one I’m starruck byan’s work.
6:18:32 · Uh so so everything from Robocop to to you know all the framing of the um the the Noble Turing challenge all of that is not only uh impressive but it’s bridging two worlds. It’s playful which is an essence of of the Japan world. So so people know about it I guess from anime maybe video games with the Nintendo side but also Sony all the all the the bridging between things.
6:19:01 · It’s very playful discovery and I think this comes from the tradition. It comes from Japan being able to mix and and people think that you know it’s an insular culture etc. It’s actually so open to well merging Buddhism branches of Daoism Shinto definitely uh and it’s not eliminating any of those.
6:19:24 · It’s maintaining diversity. How amazing is this? and it’s putting it into craft into technique into u semiconductor into designs uh architecture all of those um don’t cancel any anything else doesn’t dilute knowledge those are this is a perfect credle for those three T’s happening so I’m I’m I’m really into this we we’ll need to work together on this and uh incidentally we we also
6:19:51 · talked with Josean quite recently because he’s bridging between technology and textile, right, in a in a unique fashion. So, I I think he and a few examples there could be really the uh the sort of flagship projects that we we want to push. Yeah.
6:20:08 · Uh and Japan is the perfect credle for for uh for having this demonstrated first. Um and I guess also the apprentichip and that that’s a question I have for for for Kiosan now there’s I don’t know how much time we have but how we can create technology and this you know ACI collective intelligence that not only uh improves just that’s the danger but
6:20:33 · augments the discovery the open-endedness right so so that we can sense through like an abacus instead of a calculator so can we can we make sure that We make it this into augmenting what we have as uh as this intangible assets of Japan as an example, right? Can we take that further in augmentation not replacement?
6:20:55 · Yeah, a lot of thoughts.
6:21:01 · Uh did you have a reply at all? Uh you are what can you tell us about the link that you sent because no one else? Yeah, I just send like a link uh you know this is like fus Japan article on the sukab like whichan and I actually yeah but you know I think it’s really uh
6:21:25 · interesting uh thing is like all the locality is there and then how to actually get the uh the tradition survive uh in Japan I think that very interesting but at the same time like you want to actually reframe it you know just like you know give them uh five extra years or 10 extra years probably like it’s not going to be like a uh uh
6:21:46 · helpful I mean I it will be helpful to some extent but I think like we’re going to uh you know you know change the game you know change the I think the game book playbook need to be changed uh for that actually but at the same time like you know you know the big like a hu huge like a big trend that we are looking into is like you know that we were going
6:22:08 · to have like a AI and robotics ecosystem in a pretty much a collective intelligence in a way like a war actually will be driver system also like the way that we interact with like other people or in society will change like now these days I talk to like AI much more than humans like 90% of conversation with the AI agent and then
6:22:29 · you know and then they are very good actually I think like they I see that weakness of the AI conversation because like they are more than linear thinking very reasonable linear thinking they really fetch the uh distant like a dots like a connecting dot is still the weakness. So we have to like inject some ideas like then AI can connect like you know they can’t really search and out for like you know distant dot.
6:22:49 · So like I can see like a specific like a weakness of the transformer based architecture and then you know the strengths and the weakness but at the same time like talking to the AI is very comfortable to me because they are logically consistent and then uh rich in information and then you know they actually is very straightforward. So I think like I’m very comfortable at the same time I start seeing quite a bit of the weakness but I think at the same time like uh uh you know I’m I’m pretty sure like those weakness will be uh you know fixed.
6:23:18 · I think there will be more AI agent uh which will be more comfortable. Then uh I actually tried like a few like a stresses the personal agent but then I you know last like a few months I’ve been talking I created my you know kind of fake persona like all kind of a problem in the life and then talk to the AI about like a you know things. Then uh AI is really resonating to you.
6:23:39 · I mean this is very stressful experiments but some kind of winding down the experiment anyway but like you know I created persona which has a serious serious trouble in life and then I talk to like AI to save him or save her and you know AI is very resonating. So you know coming to the realization that the AI
6:23:57 · could be like a you know superhuman capability resonate and then you know sympathetic to you and much more than any human being you can actually think about you know if AI can be better athlete than anyone like just like you know Sony’s ace like winning a table tennis and then or like you know being a better scientist like there’s no reason to believe that the AI cannot be the best companion for you and then because like AI is like reading all email you have AI has the all the GP GPS and AI
6:24:27 · know what your boat and the Amazon and then you know with the glass AI can see what you have seen and it’s you know with the human companion that is not possible so like physically it’s not possible AI is the one actually know you most in this world and probably AI know about you much more than yourself okay then what okay then what you know this is interesting question yeah but then but you were saying you noticed there was weakness and so the question is you know it wasn’t able to go out and do that far kind of
6:24:59 · today but like people notice that like so like you know it is very obvious like I think there will be a more fixed so that there will be more