Sim-to-Real Transfer for a Robotics Task: Challenges and Lessons Learned

Reinforcement learning has been successfully applied to many robotic and non-robotic tasks in recent years. However, most of these developments have focused solely on simulated environments, eliminating safety concerns associated with a real environment and allowing for faster collection of samples. In domains such as robotics, the use of a simulation unfortunately is not sufficient due to the unpredictable, real-world effects. The trained policies need to be applicable to a non-virtual robot, which leads to further challenges for the so called domain transfer. In this paper, we present the lessons learned from training an agent-based control and transferring it to a real robot. These are: (1) the importance of sample efficiency and training times, and methods to influence them, (2) preparing the model for domain transfer, and (3) the importance of domain transfer evaluation. We propose to use a sim-to-sim transfer to thoroughly evaluate the domain transfer before integrating the agent-based control with a real robot.