PaLM-E links robot planning to broader training

Google and TU Berlin researchers submitted PaLM-E on 6 March. It feeds images and sensor information alongside language into a model that answers questions or plans robot tasks.

Photo: Anthony Quintano, “Google at the 2012 Democratic National Convention North Carolina”. Taken 6 September 2012. CC BY 2.0. Resized and cropped. Historical Google display; no endorsement implied. Not research imagery. Some rights reserved.

With 320 examples per planning task, a configuration using ViT-4B visual inputs and unchanged language-model weights improved from 30.6% and 32.9% success to 74.1% and 74.6% when trained on a broader task mixture.

Those are two simulated planning tasks, not household reliability measurements. Expert examples remained in training, and existing controllers executed the model’s text instructions.

Broader training can therefore make scarce robot demonstrations more useful. It doesn’t establish a robot that learns every physical skill without examples.