Google RT-2 brings web knowledge to robot control

Google DeepMind introduced RT-2 on 28 July 2023, showing how a robot can use knowledge learned from web images and text to interpret unfamiliar instructions. Its reported improvement comes from applying familiar physical skills to new objects and meanings, rather than learning new movements from the internet.

File photo: Google at the 2012 Democratic National Convention North Carolina, by Anthony Quintano, taken 6 September 2012. CC BY 2.0. Resized and cropped. This is a Google identity photograph, not an RT-2 demonstration.

The research connects a vision-language model directly to robot control. Where earlier systems could use a language model to plan a task for a separate controller, RT-2 trains the model itself to predict actions from camera images and instructions.

How words become movements

The research paper describes versions based on Google’s PaLI-X and PaLM-E models. Researchers continue training them on both web-based tasks and robot demonstrations. Keeping the web tasks in this stage improves generalisation compared with training on robot data alone.

For the main robot, commands become eight values: an episode-ending signal, three changes in position, three in rotation, and gripper opening. The seven continuous values are divided into 256 bins and mapped to existing tokens. During robot control, the model’s output is restricted to valid action tokens, which are decoded into movements. Fresh observations then guide the next action.

That gives web knowledge a route into the robot’s choices. Recognising a number or understanding a description can change where an object should go, while the underlying picking or placing motion still comes from robot demonstrations.

What the 62% result measures

The study reports about 6,000 evaluation trials across RT-2 and its baselines, covering familiar tasks, changed objects and surroundings, and instructions requiring semantic understanding. The main robot training data came from 13 robots working for 17 months in an office kitchen, also used for RT-1.

Table 6 reports a 62% unseen-condition average for both RT-2-PaLI-X-55B and RT-2-PaLM-E-12B, versus RT-1’s 32%. This covers easy and hard tests with unfamiliar objects, backgrounds and environments. The unseen evaluations involved more than 280 tasks focused mainly on picking and placing, not 280 newly learned physical skills. The figure isn’t a success rate for arbitrary household chores.

The breakdown matters. RT-2-PaLI-X-55B reached 96% on easier unseen backgrounds but 35% in the harder unseen environment, an office desk with monitors and accessories. Its familiar-task score was 91%, compared with RT-1’s 92%. The clearest advance is handling variation, rather than a uniform improvement in reliability.

The paper also flags that PaLM-E’s earlier training included robot images for high-level question-answering tasks that could resemble generalisation scenarios, though they contained none of the low-level actions being evaluated.

Understanding still has physical limits

A separate PaLI-3B model scored 90% in the simulated Language Table benchmark. Real-world demonstrations used that model too, but the paper records failures when unfamiliar objects behaved differently: a pen rolled away, and a banana’s weight distribution complicated pushing.

The authors also report difficulty with wiping, tool use, precise manipulation and grasping particular object parts. Selecting a rock as an improvised hammer demonstrates a semantic choice; it doesn’t establish that the robot can hammer successfully. The more elaborate planning examples come from an additionally fine-tuned PaLM-E variant and qualitative demonstrations.

Computation remains another constraint. The 55-billion-parameter PaLI-X model runs through a networked multi-TPU service at one to three control updates per second. Its 5-billion-parameter counterpart runs at around five updates per second.

The project page provides a paper and demonstrations. The launch materials don’t announce downloadable RT-2 weights or a public robot-control API.