year: 2023/12
paper: https://arxiv.org/pdf/2305.15486
website:
code: https://github.com/holmeswww/SPRING
connections: Crafter, reinforcement learning, zero-shot learning, LLM, planning, DreamerV3, Guiding Pretraining in Reinforcement Learning with Large Language Models
Problem: Open-world survival games like Crafter are challenging for AI due to complex tech trees, long-term planning needs, and multiple parallel objectives. Traditional RL methods require millions of training steps and struggle with these games.
Solution: SPRING - a novel approach that enables LLMs (specifically GPT-4) to play Crafter by:
- Reading the game’s academic paper to extract knowledge about game mechanics, objectives, and strategies
- Using structured reasoning via a directed acyclic graph (DAG) of questions to make consistent decisions
Key Innovation: Instead of training an RL agent for millions of steps, SPRING:
- Parses the LaTeX source of Crafter’s paper to understand game rules
- Converts visual game states to text descriptions
- Uses a fixed set of 9 chained questions (like “What objects do I see?”, “What are my top priorities?”, “Which action should I take?”) to reason through each game step
- Maintains consistency across hundreds of steps through structured prompting
Results:
- Zero training required (vs 1M+ steps for RL methods)
- 88% better game score than previous state-of-the-art (DreamerV3)
- Achieves 10x higher success rates on difficult tasks like crafting stone tools
- Outperforms all RL baselines including DreamerV3, PPO, and Rainbow