Optimal decision making, at its core, requires considering counterfactuals.
If I did X instead of Y, would it be better? You can answer that question using a learned simulator, a value function, a reward model, … in the end it’s kinda all the same, as long as you have some mechanism for figuring out which counterfactual is better.
→ The key is not necessarily to do really good simulations, but how to answer counterfactuals.Simulation through a learned model seems to be the way the brain figures out counterfactuals (mostly during sleep).
The ability to utilize prior knowledge is the key enabler for generalization and progress in AI (… culture, more broadly?)
inattentional blindness in humans focussing on a task, robots vision and repr will be tractable? becuase they have this focusing mechanism bcs they want to achievr a goal.. as opposed to like going super in depth on how clouds work when its looking out the window
compositional generalization ⇒ “emergent capabilities” (a la ICL)
if you have good stepping stones, it’s easy to get surprising new capabilities
LLM representations being entangled points to a lot of room for efficiency gains on that front
→ “It’s all about representations”