Reinforcement Learning - from Fundamentals to Deep RL
Description
Our training provides an overview of Reinforcement Learning. We start from the mathematics needed, through basic RL algorithms, and get into Deep Reinforcement Learning and the latest state of the art methods used today. We go through some applications in detail and also cover some significant achievements in the field so far.
The course focuses on some of the main issues which arise when dealing with RL in the real world, and goes through some of the main algorithms which are the backbone of newer RL systems.
Significant theoretical knowledge is gained, not only for RL but for ML in general, with practical applications throughout. From the basics of Linear Algebra, Calculus, and Probabilities, we go through Dynamic Programming and Markov Processes, to finally reach the ubiquitous Q-learning and its deep variants, as well as some policy gradient methods.
We strive to give a comprehensive overview, at least with respect to different fundamental techniques employed in the literature. The second half of the course is only about RL with neural networks, with research papers discussed in detail and various applications explained.
Practical tools are also discussed and used in exercises (from Pytorch to Ray).
is issued on the Luxoft Training form
Objectives
- Understand the main modelling difficulties in developing RL algorithms
- Learn how to use existent algorithms and understand issues which arise in state representation or reward shaping
- Visualize and adapt the algorithm or reward mechanism, so that the agent learns one thing or another
- Be able to apply or develop RL algorithms for real-life problems
Target Audience
- Software Developers who are knowledgeable about Python/Machine Learning, yet lack experience in Reinforcement Learning
- Machine Learning Engineers
- Data Scientists
Prerequisites
- Graduate level of calculus, probability theory, discrete mathematics
- Basic knowledge of machine learning
- Ability to understand Python code
Roadmap
Linear algebra
- Overview of concepts
Calculus
- Integration
- Derivation
- Examples
Probabilities
- Random variables
- Density functions
- Expectation
- Conditional, joint and marginal probabilities
- Examples
- Practice
RL Introduction
- Markov Decision Processes
- Dynamic programming with example
- Bellman equation
- Policy evaluation
- Policy Iteration
- Value Iteration
- Examples
- Practice
Model-based vs model-free
- Learning and planning
- Deterministic
- Stochastic
- Linear value-function approximation
- Comparison and practice
Algorithms
- Q-learning
- Sarsa
- Actor-critic
- Policy gradient
- Monte-Carlo tree methods
- Exploration vs exploitation
- Examples
- Practice
Deep Reinforcement Learning
- Nonlinear function approximation
- The DeepMind breakthrough
- Alpha-Star Explained
Latest technologies in DRL
- Memory, Attention, Recurrence
- Inverse RL
- Multi-agent
- Hierarchical
- Evolved rewards – AutoRL
- Policy optimization
Applications and use
- Trading
- Speech understanding and question answering (optional)
- Load balancing (optional)
- Other uses (optional)
Pytorch / Tensorflow
- Tensor basics
- Implementing a RL algorithm from scratch
- Testing and visualizing
- Practice
Ray + RLlib
- Main concepts: actors, futures, memory sharing, etc.
- Worked example
- Different algorithms
- Grid search and visualization
- Practice
Visualization and explainability
- SMDP, AMDP, SAMDP
- Projection to 3D-space with TSNE
- Examples
"TYPE"html";}