Awesome Open Source
Awesome Open Source

Practical_RL

An open course on reinforcement learning in the wild. Taught on-campus at HSE and YSDA and maintained to be friendly to online students (both english and russian).

Manifesto:

  • Optimize for the curious. For all the materials that aren’t covered in detail there are links to more information and related materials (D.Silver/Sutton/blogs/whatever). Assignments will have bonus sections if you want to dig deeper.
  • Practicality first. Everything essential to solving reinforcement learning problems is worth mentioning. We won't shun away from covering tricks and heuristics. For every major idea there should be a lab that makes you to “feel” it on a practical problem.
  • Git-course. Know a way to make the course better? Noticed a typo in a formula? Found a useful link? Made the code more readable? Made a version for alternative framework? You're awesome! Pull-request it!

Github contributors

Course info

Additional materials

Syllabus

The syllabus is approximate: the lectures may occur in a slightly different order and some topics may end up taking two weeks.

  • week01_intro Introduction

    • Lecture: RL problems around us. Decision processes. Stochastic optimization, Crossentropy method. Parameter space search vs action space search.
    • Seminar: Welcome into openai gym. Tabular CEM for Taxi-v0, deep CEM for box2d environments.
    • Homework description - see week1/README.md.
  • week02_value_based Value-based methods

    • Lecture: Discounted reward MDP. Value-based approach. Value iteration. Policy iteration. Discounted reward fails.
    • Seminar: Value iteration.
    • Homework description - see week2/README.md.
  • week03_model_free Model-free reinforcement learning

    • Lecture: Q-learning. SARSA. Off-policy Vs on-policy algorithms. N-step algorithms. TD(Lambda).
    • Seminar: Qlearning Vs SARSA Vs Expected Value SARSA
    • Homework description - see week3/README.md.
  • recap_deep_learning - deep learning recap

    • Lecture: Deep learning 101
    • Seminar: Intro to pytorch/tensorflow, simple image classification with convnets
  • week04_approx_rl Approximate (deep) RL

    • Lecture: Infinite/continuous state space. Value function approximation. Convergence conditions. Multiple agents trick; experience replay, target networks, double/dueling/bootstrap DQN, etc.
    • Seminar: Approximate Q-learning with experience replay. (CartPole, Atari)
  • week05_explore Exploration

    • Lecture: Contextual bandits. Thompson Sampling, UCB, bayesian UCB. Exploration in model-based RL, MCTS. "Deep" heuristics for exploration.
    • Seminar: bayesian exploration for contextual bandits. UCB for MCTS.
  • week06_policy_based Policy Gradient methods

    • Lecture: Motivation for policy-based, policy gradient, logderivative trick, REINFORCE/crossentropy method, variance reduction(baseline), advantage actor-critic (incl. GAE)
    • Seminar: REINFORCE, advantage actor-critic
  • week07_seq2seq Reinforcement Learning for Sequence Models

    • Lecture: Problems with sequential data. Recurrent neural networks. Backprop through time. Vanishing & exploding gradients. LSTM, GRU. Gradient clipping
    • Seminar: character-level RNN language model
  • week08_pomdp Partially Observed MDP

    • Lecture: POMDP intro. POMDP learning (agents with memory). POMDP planning (POMCP, etc)
    • Seminar: Deep kung-fu & doom with recurrent A3C and DRQN
  • week09_policy_II Advanced policy-based methods

    • Lecture: Trust region policy optimization. NPO/PPO. Deterministic policy gradient. DDPG
    • Seminar: Approximate TRPO for simple robot control.
  • week10_planning Model-based RL & Co

    • Lecture: Model-Based RL, Planning in General, Imitation Learning and Inverse Reinforcement Learning
    • Seminar: MCTS for toy tasks
  • yet_another_week Inverse RL and Imitation Learning

    • All that cool RL stuff that you won't learn from this course :)

Course staff

Course materials and teaching by: [unordered]

Contributions

Alternatives To Practical_rl
Select To Compare


Alternative Project Comparisons
Related Awesome Lists
Top Programming Languages

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Jupyter Notebook (179,412
Learning (75,687
Deep Learning (39,340
Tensorflow (22,900
Pytorch (22,655
Course (20,513
Keras (10,612
Reinforcement Learning (4,882
Rl (2,335
Deep Reinforcement Learning (1,116
Course Materials (775
Mooc (667
Pytorch Tutorial (321
Git Course (3