On the Variance of the Adaptive Learning Rate and Beyond

Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|

Pytorch Optimizer | 2,658 | 23 | 2 months ago | 21 | October 31, 2021 | 41 | apache-2.0 | Python | ||

torch-optimizer -- collection of optimizers for Pytorch | ||||||||||

Radam | 2,393 | 2 years ago | 11 | apache-2.0 | Python | |||||

On the Variance of the Adaptive Learning Rate and Beyond | ||||||||||

Yellowfin_pytorch | 293 | 4 years ago | 7 | apache-2.0 | Python | |||||

auto-tuning momentum SGD optimizer | ||||||||||

Emotion Rating View | 289 | 4 years ago | apache-2.0 | Kotlin | ||||||

A library for animated rating view in Android apps. | ||||||||||

Adabound Tensorflow | 135 | 4 years ago | 5 | apache-2.0 | Python | |||||

Simple Tensorflow implementation of "Adaptive Gradient Methods with Dynamic Bound of Learning Rate" (ICLR 2019) | ||||||||||

Android Ratingreviews | 103 | 2 years ago | 1 | February 25, 2019 | 9 | apache-2.0 | Java | |||

Simple star rating system bars, a view similar to the ones seen on Google Playstore. ⭐🌟✨ | ||||||||||

Hypergradient Descent | 95 | 4 years ago | 7 | mit | Python | |||||

Hypergradient descent | ||||||||||

Backpack Android | 94 | 2 days ago | 22 | June 17, 2022 | 7 | apache-2.0 | Kotlin | |||

Backpack Design System | ||||||||||

Cocob | 80 | 6 years ago | 2 | apache-2.0 | Python | |||||

TensorFlow implementation of COCOB | ||||||||||

Neural Net Optimization | 49 | 3 years ago | mit | Python | ||||||

PyTorch implementations of recent optimization algorithms for deep learning. |

Alternatives To RadamSelect To Compare

Alternative Project Comparisons

Readme

We are in an early-release beta. Expect some adventures and rough edges.

- Introduction
- Motivation
- Questions and Discussions
- Quick Start Guide
- Related Posts and Repos
- Citation

The learning rate warmup for Adam is a must-have trick for stable training in certain situations (or eps tuning). But the underlying mechanism is largely unknown. In our study, we suggest one fundamental cause is **the large variance of the adaptive learning rates**, and provide both theoretical and empirical support evidence.

In addition to explaining **why we should use warmup**, we also propose **RAdam**, a theoretically sound variant of Adam.

As shown in Figure 1, we assume that gradients follow a normal distribution (mean: \mu, variance: 1). The variance of the adaptive learning rate is simulated and plotted in Figure 1 (blue curve). We observe that the adaptive learning rate has a large variance in the early stage of training.

When using a Transformer for NMT, a warmup stage is usually required to avoid convergence problems (e.g., Adam-vanilla converges around 500 PPL in Figure 2, while Adam-warmup successfully converges under 10 PPL). In further explorations, we notice that, if we use additional 2000 samples to estimate the adaptive learning rate, the convergence problems are avoided (Adam-2k); or, if we increase the value of eps, the convergence problems are also relieved (Adam-eps).

Therefore, we conjecture that the large variance in the early stage causes the convergence problem, and further propose Rectified Adam by analytically reducing the large variance. More details can be found in our paper.

Yes, the robustness of RAdam is not infinity. In our experiments, it works for a broader range of learning rates, but not all learning rates.

**Choice of the Original Transformer.** We choose the original Transformer as our main study object because, without warmup, it suffers from the most serious convergence problems in our experiments. With such serious problems, our controlled experiments can better verify our hypothesis (i.e., we demonstrate that Adam-2k / Adam-eps can avoid spurious local optima by minimal changes).

**Sensitivity.** We observe that the Transformer is sensitive to the architecture configuration, despite its efficiency and effectiveness. For example, by changing the position of the layer norm, the model may / may not require the warmup to get a good performance. Intuitively, since the gradient of the attention layer could be more sparse and the adaptive learning rates for smaller gradients have a larger variance, they are more sensitive. Nevertheless, we believe this problem deserves more in-depth analysis and is beyond the scope of our study.

Although the adaptive learning rate has a larger variance in the early stage, the exact magnitude is subject to the model design. Thus, the convergent problem could be more serious for some models/tasks than others. In our experiments, we observe that RAdam achieves consistent improvements over the vanilla Adam. It verifies the variance issue widely exists (since we can get better performance by fixing it).

As in Figure 1 (above), even if the gradient is not zero-meaned, the original adaptive learning rate still has a larger variance in the beginning, thus applying the rectification can help to stabilize the training.

Another related concern is that, when the mean of the gradient is significantly larger than its variance, the magnitude of the "problematic" variance may not be very large (i.e., in Figure 1, when \mu equals to 10, the adaptive learning rate variance is relatively small and may not cause problems). We think it provides a possible explaination on why warmup have a bigger impact on some models than others. Still, we suggest that, in real-world applications, neural networks usually have some parts of parameters meet our assumption well (i.e., their gradient variance is larger than their gradient mean), and needs the rectification to stabilize the training.

To the best of our knowledge, the warmup heuristic is originally designed for large minibatch SGD [0], based on the intuition that the network changes rapidly in the early stage. However, we find that it **does not** explain why Adam requires warmup. Notice that, Adam-2k uses the same large learning rate but with a better estimation of the adaptive learning rate can also avoid the convergence problems.

The reason why sometimes warmup also helps SGD still lacks of theoretical support. FYI, when optimizing a simple 2-layer CNN with gradient descent, the thoery of [1] could be used to show the benifits of warmup. Specifically, the lr must be $O(cos \phi)$, where $\phi$ is the angle between the current weight and the ground true weight and $cos \phi$ could be very small due to high dimensional space and random initialization. And thus lr must be very small at the beginning to guarentee the convergence. $cos \phi$ however can be improved in the later stage, and thus the learning rate is allowed to be larger. Their theory somehow can justify why warmup is needed by gradient descend and neural networks. But it is still far-fetched for the real scenario.

[0] Goyal et al, Accurate, Large Minibatch SGD: Training Imagenet in 1 Hour, 2017

[1] Du et al, Gradient Descent Learns One-hidden-layer CNN: Don’t be Afraid of Spurious Local Minima, 2017

- Directly replace the vanilla Adam with RAdam without changing any settings.
- Further tune hyper-parameters (including the learning rate) for a better performance.

Note that in our paper, our major contribution is **to identify why we need the warmup for Adam**. Although some researchers successfully improve their model performance (**user comments**), considering the difficulty of training NNs, directly plugging in RAdam **may not** result in an immediate performance boost. Based on our experience, replacing **the vanilla Adam** with RAdam usually results in a better performance; however, if **warmup has already been employed and tuned** in the baseline method, it is necessary to also tune hyper-parameters for RAdam.

RAdam is very easy to implement, we provide PyTorch implementations here, while third party ones can be found at:

Julia implementation in Flux.jl

We provide a simple introduction in Motivation, and more details can be found in our paper. There are some unofficial introductions available (with better writings), and they are listed here for reference only (contents/claims in our paper are more accurate):

We are happy to see that our algorithms are found to be useful by some users : -)

"...I tested it on ImageNette and quickly got new high accuracy scores for the 5 and 20 epoch 128px leaderboard scores, so I know it works... https://forums.fast.ai/t/meet-radam-imo-the-new-state-of-the-art-ai-optimizer/52656

— Less Wright August 15, 2019

Thought "sounds interesting, I'll give it a try" - top 5 are vanilla Adam, bottom 4 (I only have access to 4 GPUs) are RAdam... so far looking pretty promising! pic.twitter.com/irvJSeoVfx

— Hamish Dickson (@_mishy) August 16, 2019

RAdam works great for me! It’s good to several % accuracy for free, but the biggest thing I like is the training stability. RAdam is way more stable! https://medium.com/@mgrankin/radam-works-great-for-me-344d37183943

— Grankin Mikhail August 17, 2019

"... Also, I achieved higher accuracy results using the newly proposed RAdam optimization function.... https://towardsdatascience.com/optimism-is-on-the-menu-a-recession-is-not-d87cce265b10

— Sameer Ahuja August 24, 2019

"... Out-of-box RAdam implementation performs better than Adam and finetuned SGD... https://twitter.com/ukrdailo/status/1166265186920980480

— Alex Dailo August 27, 2019

Please cite the following paper if you found our model useful. Thanks!

Liyuan Liu , Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han (2020). On the Variance of the Adaptive Learning Rate and Beyond. the Eighth International Conference on Learning Representations.

```
@inproceedings{liu2019radam,
author = {Liu, Liyuan and Jiang, Haoming and He, Pengcheng and Chen, Weizhu and Liu, Xiaodong and Gao, Jianfeng and Han, Jiawei},
booktitle = {Proceedings of the Eighth International Conference on Learning Representations (ICLR 2020)},
month = {April},
title = {On the Variance of the Adaptive Learning Rate and Beyond},
year = {2020}
}
```

Popular Rating Projects

Popular Gradient Projects

Popular User Interface Components Categories

Related Searches

Get A Weekly Email With Trending Projects For These Categories

No Spam. Unsubscribe easily at any time.

Python

Rate

Gradient