Bi-Phase Episodic Memory-Guided Deep Reinforcement Learning for Robot Skills

Deep reinforcement learning has significant advantages in the field of robot skill learning, however it usually cannot make good use of experience. In this letter, a Bi-phase Episodic Memory Guided (BEMG) method for speeding up the learning process of DRL and solving the problem of sparse reward is proposed, which brings a large learning speed increasing of robotic operation skills acquisition through memory guidance. The method constructs the relationship between robot states at different moments through the guidance of episodic memory, and automatically generates reward function through memory backtracking. In the framework of proposed method, each iteration is divided into two phases. In first phase, the association between state and action in memory is used to guide the action decision-making of DRL, thereby reducing unnecessary exploration for learning. In second phase, the memory module automatically generates reward values based on the relationship between different states, instead of artificially designing reward functions. The experimental verifications are carried out around robot operation skill learning tasks with different DRL algorithms. Experimental results show the proposed method can effectively improve the learning speed of robot operation skills and avoid the dependence on complex reward function.