Reinforcement Learning - from Fundamentals to Deep RL

Reinforcement Learning - From Fundamentals to Deep RL is an extensive course designed to provide a comprehensive understanding of reinforcement learning, from its theoretical foundations to the latest advancements in deep reinforcement learning. This course is ideal for graduate-level students, data scientists, AI researchers, and professionals looking to expand their knowledge and apply RL in various domains.

The course begins with a solid background in essential mathematical concepts, including linear algebra, calculus, and probability theory, ensuring you have the necessary tools to tackle more advanced topics. You’ll gain a deep understanding of random variables, density functions, expectation, and probabilities, setting a strong foundation for the rest of the course.

Next, you’ll explore the core concepts of reinforcement learning, including Markov Decision Processes (MDPs), dynamic programming, and key algorithms like Q-learning, Sarsa, and Actor-Critic. You’ll learn the differences between model-based and model-free approaches and how to implement these techniques in both deterministic and stochastic environments. This section of the course includes extensive examples and hands-on practice to reinforce your understanding.

In the deep reinforcement learning section, you’ll delve into advanced topics such as nonlinear function approximation, breakthroughs from DeepMind, and cutting-edge technologies like memory, attention, recurrence, and hierarchical learning. You’ll also explore practical applications of deep RL in trading, speech understanding, load balancing, and more.

The course concludes with a focus on practical tools and implementations. You’ll work with PyTorch and TensorFlow to implement RL algorithms from scratch, use Ray and RLlib for scalable reinforcement learning applications, and learn how to visualize and explain RL models using advanced techniques like TSNE.

With a balanced approach between theory (24.5 hours) and practice (14.5 hours), this course ensures you not only understand the concepts but also gain the hands-on experience needed to apply RL in real-world scenarios.

By the end of this course, participants will:

• Understand the mathematical foundations required for reinforcement learning.
• Master core RL concepts, including Markov Decision Processes, policy evaluation, and key algorithms like Q-learning and Actor-Critic.
• Implement deep reinforcement learning techniques and explore the latest advancements in the field.
• Apply RL in practical domains such as trading, speech understanding, and load balancing.
• Use tools like PyTorch, TensorFlow, Ray, and RLlib to build and deploy RL models at scale.

This course offers a comprehensive mix of theory and practice, providing a deep dive into reinforcement learning and its applications. Through a series of practical exercises, you’ll gain the skills and experience needed to implement RL models and apply them in real-world projects.

Upon completion of the "Reinforcement Learning - From Fundamentals to Deep RL" course, trainees will be able to:

• Grasp the fundamental mathematical concepts necessary for reinforcement learning.
• Apply core RL algorithms and techniques in both deterministic and stochastic environments.
• Explore and implement advanced deep reinforcement learning models and technologies.
• Utilize modern tools and frameworks like PyTorch, TensorFlow, and Ray to develop and deploy scalable RL solutions.
• Visualize and explain RL models using advanced techniques to ensure interpretability and transparency.

Part I. Background (5h30m)

      1. Linear algebra (30m)

                             a. Overview of concepts

              2. Calculus (1h)

                             a. Integration

                             b. Derivation

                             c. Examples

              3. Probabilities (4h)
                             a. Random variables

                             b. Density functions

                             c. Expectation

                             d. Conditional, joint and marginal probabilities

                             e. Examples
                             f. Practice

Part II. Overview (11h)

              1. RL Introduction (4h)

                             a. Markov Decision Processes

                             b. Dynamic programming with example

                             c. Bellman equation

                             d. Policy evaluation

                             e. Policy Iteration

                             f. Value Iteration

                             g. Examples

                             h. Practice
              2. Model-based vs model-free (3h)

                             a. Learning and planning

                             b. Deterministic

                             c. Stochastic
                             d. Linear value-function approximation
                             e. Comparison and practice
              3. Algorithms (4h)
                             a. Q-learning
                             b. Sarsa
                             c. Actor-critic
                             d. Policy gradient
                             e. Monte-Carlo tree methods

                             f. Exploration vs exploitation

                             e. Examples
                             g. Practice

Part III. RL + Deep Learning (6h30m)

              1. Deep Reinforcement Learning
                             a. Nonlinear function approximation
                             b. The DeepMind breaktrough
                             c. Alpha-Star Explained
              2. Latest technologies in DRL (3h)
                             a. Memory, Attention, Recurrence
                             b. Inverse RL
                             c. Multi-agent

                             d. Hierarchical
                             e. Evolved rewards – AutoRL

                             f. Policy optimization

              3. Applications and use (2h):

                             a. Trading
                             b. Speech understanding and question answering (optional)
                             c. Load balancing (optional)
                             d. Other uses (optional)
Part IV. Practical examples and tools (7h)

              1. Pytorch / Tensorflow (2h)
                             a. Tensor basics
                             b. Implementing a RL algorithm from scratch
                             c. Testing and visualizing

                             d. Practice

              2. Ray + RLlib (3h30)

                             a. Main concepts: actors, futures, memory sharing, etc.
                             b. Worked example
                             c. Different algorithms
                             d. Grid search and visualization
                             e. Practice

3. Visualization and explainability (1h30)

                             a. SMDP, AMDP, SAMDP

                             b. Projection to 3D-space with TSNE

                             c. Examples