Open X-Embodiment

The model takes as input a RGB image from the robot workspace camera and a task string describing the task that the robot is supposed to perform.

What task the model should perform is communicated to the model purely through the task string. The image communicates to the model the current state of the world, i.e. assuming the model runs at three hertz, every 333 milliseconds, we feed the latest RGB image from a robot workspace camera into the model to obtain the next action to take.