Deep Neural Networks and Experience Replay in Q-learning Extensions: A Review

Reinforcement Learning (RL) is a type of machine learning where actions are learned and taken to solve sequential decision problems. There have been several extensions in the Q-learning algorithm, but more extensions have occurred since the breakthrough of the Deep Q-network algorithm. We study the two pivotal aspects of the DQN algorithm (deep neural network and experience replay) and other related extensions; we focus on experience replay. Our study identifies multiple extensions in network structure, experience sampling strategies, memory managing techniques, and memory structures. We further indicate the extended algorithms’ strengths and weaknesses and suggest future works.