2D1431 Machine Learning Lab 3: Reinforcement Learning

In this lab you will use reinforcement learning to search for a working control algorithm for a simple walking robot. We will assume that you are familiar with the basic concepts of reinforcement learning and that you have read chapter 13 in the course book Machine Learning (Mitchell, 1997). As supplementary material you may read the first four chapters of the survey on reinforcement learning by Kaelbling et al. (1996). For further reading and a detailed discussion of policy iteration and reinforcement learning, the textbook “Reinforcement Learning” is highly recommended (Sutton and Barto, 1999). Policy iteration is an algorithm which can be used to compute optimal policies given a model of the environment. Policy iteration solves problems that can be formulated as Markov decision processes, i.e. when the state transitions and reward functions are known beforehand, and utilizes the principles of dynamic programming. When the environment is unknown and only accessible by interacting with it, the agent will have to use learning to solve the problem. Temporal difference learning is a technique which resembles policy iteration but without the need for a known environment. The central idea of both policy iteration and temporal difference learning is to estimate value functions, which in turn can be used to identify the optimal policy. The value function is essential in that it makes the value of predicted future events accessible to the process which decides what to do now. Reinforcement based methods are therefore able to do a basic form of planning.