https://www.dwarkesh.com/p/ilya-sutskever-2
Ilya sutskever dwarkesh interview things that popped into my mind:
- humans are incredibly robust → voting mechanism (c.f. ensemble of models) leads to robustness (variance reduction)
- he too brought up neurons potentially being much more complex than we think [… each being their own agent] (“and if that turns out to play and important role, things might turn out to be much more complex”).
Other interesting bits:
- has many opinions on this but can’t talk
- they have comparable amounts of compute for research than frontier labs (3b raised)
- calls “superintelligence” an (efficient) (continual) learning algorithm that allows a mind to learn anything/everything
- with this type of AI, very rapid economic growth might be possible (given little regulations and other real-world frictions)
- →We want to build (democratic) AI that’s robustly aligned to care about sentient life specifically.
- after all, we model others with the same circuit that we use to model ourselves, because that’s the most efficient thing to do
- his timeline for such a system, that can learn as well as a human and subsequently becomes superhuman is 5-20 years
- he is bearish on current frontier companies achieving this in the short term, but thinks the technical approach (as well as the perspective on alignment as stated above) to converge.
- an argument for a good world is that a company that reaches asi first can’t capture everything (since combinatorial explosion of like task space, environments, goals, etc. as technical capabilities grow, as has been happening ever since), and that this will lead to niching/specialization of different asi systems, which will then have to cooperate to achieve more complex goals (… but he doesn’t think this is how it’s going to turn out (why?)).
- all pretrained llms are the same because… the data is the same; differentiation via RL on different tasks
- Research taste: “It’s beauty, simplicity, elegance, correct inspiration from the brain. All of those things need to be present at the same time. The more they are present, the more confident you can be in a top-down belief.”
- We’re transitioning from scaling (‘20-‘25) to research again, until we find the next sigmoid to scale on. Good ideas are + hands to test them out are the bottleneck, not compute anymore.
From https://x.com/kjaved_/status/1993434387132613030:
Intelligence is about the ability to learn and not about knowing many things. The right goal is a system that can learn from experience in deployment.
A value function is needed for human-like sample-efficient learning. It can provide dense feedback (TD learning) in the absence of reward.
Both of these are essential and doable. A key bottleneck is that we don’t have algorithms that can learn reliably using similar amounts of compute as inference. Such algorithms are needed if we are to learn continually.
I think we are close. We just don’t have enough people working on finding these algorithms.