Active Predictive Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problems

In this article, we propose a backpropagation-free approach to robotic control through the neuro-cognitive computational framework of neural generative coding (NGC), designing an agent completely built from predictive processing circuits that facilitate dynamic, online learning from sparse rewards, embodying the principles of planning-as-inference. Concretely, we craft an adaptive agent system, which we call active predictive coding (ActPC), that balances an internally-generated epistemic signal (meant to encourage intelligent exploration) with an internally-generated instrumental signal (meant to encourage goal-seeking behavior) to learn how to control various simulated robotic systems as well as a complex robotic arm using a realistic simulator, i.e., the Surreal Robotics Suite, for the block lifting task and the can pick-and-place problem. Notably, our results demonstrate that the proposed ActPC agent performs well in the face of sparse (extrinsic) reward signals and is competitive with or outperforms several powerful backpropagation-based reinforcement learning approaches.