Overview

Perturbations applied to various tasks from the benchmark.

Introduction

Colosseum is a benchmark that test the generalization capabilities of robot manipulation policies. It does so in a similar approach to Xie et al. [XLXF23] and Zhu et al. [ZJSZ23] by varying environmental factors that can affect generalization of robot manipulation policies. Our simulated benchmark builds on top of RLBench (James et al. [JMRAD20]) by defining 12 perturbation factors that the user can control and collect demonstrations for training and testing policies under these variations.

Below we list the perturbation factors and give a brief description. Note that we make use of the following categorization:

• Manipulation Object (MO) perturbation : MO is a task-relevant object that is directly manipulated or interacted with by the robot.

• Receiver Object (RO) perturbation: RO is a task-relevant object that is not directly interacted with by the robot.

• Background perturbation: Factors that do not relate to task-relevant objects, but are background characteristics of the scene.

Factor of Variation	Description
MO color	Modifies the color of the MO
RO color	Modifies the color of the RO (if applicable)
MO texture	Modifies the texture applied to the MO
RO texture	Modifies the texture applied to the RO (if applicable)
MO size	Scales the MO by a given factor
RO size	Scales the RO (if applicable) by a given factor
Light color	Modifies the color of the lights setup in the scene.
Table color	Modifies the color of the tabletop of the robot setup
Table texture	Modifies the texture applied to the tabletop of the robot setup.
Distractor object	Spawns a random object in the workspace of the robot.
Background texture	Modifies the textures applied to the walls of the scene.
Camera pose	Randomly perturbs the pose of a camera.

Perturbations

1. MO Color

Task Open Drawer with no variations

Task Open Drawer with object color variation applied to drawer

2. RO Color

Task Basketball In Hoop with no variations

Task Basketball In Hoop with object color variation applied to hoop

3. MO Texture

Task Basketball In Hoop with no variations

Task Basketball In Hoop with object texture variation applied to ball

4. RO Texture

Task Basketball In Hoop with no variations

Task Basketball In Hoop with object texture variation applied to ball

5. MO Size

Task Close Box with no variations

Task Close Box with object size variation applied to the box

6. RO Size

Task Hokey with no variations

Task Hockey with object size variation applied to the ball

7. Light Color

Task Reach and Drag with no variations

Task Reach and Drag with light color variation applied

8. Table Color

Task Hockey with no variations

Task Hockey with table color variation applied to the tabletop

9. Table Texture

Task Meat On Grill with no variations

Task Meat On Grill with table texture variation applied to the tabletop

10. Distractor Object

Task Insert onto square peg with no variations

Task Insert onto square peg with distractor object variation being applied

11. Background Texture

Task Basketball In Hoop with no variations

Task Basketball In Hoop with background texture variation being applied

12. Camera Pose

Task Setup Chess with no variations

Task Setup Chess with camera pose variation being applied

JMRAD20

Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: the robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020.

XLXF23

Annie Xie, Lisa Lee, Ted Xiao, and Chelsea Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. 2023. arXiv:2307.03659.

ZJSZ23

Yifeng Zhu, Zhenyu Jiang, Peter Stone, and Yuke Zhu. Learning generalizable manipulation policies with object-centric 3d representations. In 7th Annual Conference on Robot Learning. 2023.