CMPUT 652: Reinforcement Learning with Robots (Fall 2019)


In this graduate course, students learn how to develop control methods that they can evaluate in their own created worlds by understanding the fundamentals of MDPs, iterative methods, stochastic approximation methods and policy gradient methods. Then they apply their developed methods to learn to control physical robots. Enroute, they develop tools and understanding for using physical robots nearly as easily as the simulated ones.

DXL-Tracker UR-Reacher-6 Create-Docker


[Schedule] (Contains slides and course materials)

Syllabus

Overview

In this new course, we will study how reinforcement learning (RL) algorithms can be used to learn to control physical robots in real-time. One of the main goals of RL agents is to learn to solve a given task by interacting with an unknown, unstructured environment. Due to simple and intuitive foundations and a series of breakthroughs in computer and board games, RL has become one of the most appealing branches of artificial intelligence. Recent RL methods have also shown excellent results in controlling robots in the virtual world. Now RL is viewed as a promising approach for controlling and adapting to unstructured environments with physical robots. To what extent are current methods capable of achieving this goal?

We will study the foundations of RL to be able to develop policy learning methods and learn about systematic ways of studying a real-time system to reveal the uncertainties involved in real-world tasks. This investigation will allow us to understand the differences between real-world and standard simulated tasks so that we can adapt task setups and algorithmic implementations to the real world as well as enhance the simulated tasks to incorporate the additional challenges in real-time systems. En route, we will learn about other promising approaches to learning in robotics that are not performed in real-time, such as learning from demonstration and simulation-to-reality transfer.

Background in machine learning, programming, or robotics would be essential in this course.

Objectives

Prerequisites

This course requires knowledge in basic probability theory and programming ability using deep learning frameworks such as PyTorch in Python 3.

Course Materials

The book by Sutton and Barto (2018) will be used as the main textbook. Apart from that, we will rely on scientific papers, which will be provided in the course schedule.

Course Topics

Course work and evaluation