GEN-1.5 learns physical tasks from a 3-second demonstration without training
Generalist AI's GEN-1.5 model can learn a new robot task within seconds from a single 3–12 second demonstration without additional training. Built on large-scale physical interaction data, the model achieves 59% success with one-shot in-context learning and 83% with few-shot learning, while spontaneously exhibiting zero-shot sim-to-real transfer, compositional chaining of behaviors, and tool improvisation capabilities.
The physical world is one of the greatest challenges in robotics and AI. If we manage to overcome it, we reach a state where AI-powered robots can do our work or collaborate with us in the real world.
How does one-shot learning in robotics differ from one-shot learning in large language models?
Language models like GPT-3 operate on symbolic representations, making one-shot learning of text tasks more tractable. Robotics requires the model to handle rich sensory feedback, physical dynamics, and real-world environmental variation in a closed-loop setting, making the task substantially more complex.
What is "physical prompting" and how does it differ from traditional fine-tuning in robotics?
Physical prompting involves placing video or sensorimotor sequences from one or more demonstrations into the model's context window. Unlike traditional fine-tuning, which updates model weights through gradient descent on a training dataset, physical prompting achieves adaptation without any gradient updates.
Which capabilities emerged in GEN-1.5 without explicit training signals?
Without special architectural design or meta-learning loops, the model spontaneously developed abilities including chaining multiple physical prompts into longer behaviors, transferring demonstrations from simulation to the real world, and imitating human hand gestures with robotic arms. These emerged only from pretraining on large quantities of physical interaction data.
- An Alien Mind: OpenAI's Chief Scientist on Intelligence We Don't Fully Understand — openai.com 75 % match
- Understanding is the new bottleneck — geoffreylitt.com 73 % match
- How to Build a Diffusion Language Model — kuleshov-group.github.io 72 % match
- GEN-1.5
- Generalist AI
- GPT-3