- This repository includes a Keras callback to be used in training that allows implementation of cyclical learning rate policies, as detailed in Leslie Smith’s paper Cyclical Learning Rates for Training Neural Networks arXiv:1506.01186v4.
- When you scale on cycle, the absolute change in learning rate from one iteration to the next is always constant in a cycle.
- This allows you to plot your learning rates over training iterations, even after you change/reset the cycle.
- The author offers a simple approach to determining the boundaries of your cycle by increasing the learning rate over a number of epochs and observing the results.
- In order to plot accuracy vs learning rate, you can use the attribute to get the learning rates and accuracy at each iteration.

Contribute to CLR development by creating an account on GitHub.

@jimmfleming: *Cool library for cyclic learning rates in Keras: #deeplearning*

This repository includes a Keras callback to be used in training that allows implementation of cyclical learning rate policies, as detailed in Leslie Smith’s paper Cyclical Learning Rates for Training Neural Networks arXiv:1506.01186v4.

A cyclical learning rate is a policy of learning rate adjustment that increases the learning rate off a base value in a cyclical nature. Typically the frequency of the cycle is constant, but the amplitude is often scaled dynamically at either each cycle or each mini-batch iteration.

clr policy (detailed below). With clr, their model reached 81.4% validation accuracy in only 25,000 iterations compared to 70,000 iterations with standard hyperparameter settings.

One reason this approach may work well is because increasing the learning rate is an effective way of escaping saddle points. By cycling the learning rate, we’re guaranteeing that such an increase will take place if we end up in a saddle point.

CyclicLR()

The purpose of this class is to not only provide an easy implementation of CLR for Keras, but to enable easy experimentation with policies not explored in the original paper.

, as detailed in the original paper. It also allows for custom amplitude scaling functions, enabling easy experimentation.

Arguments for this class include:

The general structure of the policy algorithm is:

can be used with any optimizer in Keras.

Syncing cycle and training iterations

The author points out that the best accuracies are typically attained by ending with the base learning rate. Therefore it’s recommended to make sure your training finishes at the end of the cycle.

triangular

This method is a simple triangular cycle.

Basic algorithm:

Default triangular clr policy example:

Results:

triangular2

This method is a triangular cycle that decreases the cycle amplitude by half after each period, while keeping the base lr constant. This is an example of scaling on cycle number.

Basic algorithm:

Default triangular clr policy example:

Results:

, while keeping the base lr constant. This is an example of scaling on iteration.

Basic algorithm:

Default triangular clr policy example:

Results:

This method is a triangular cycle that scales the cycle amplitude sinusoidally. This is an example of scaling on cycle.

Basic algorithm:

Default custom cycle-policy example:

Results:

This method is a triangular cycle that scales the cycle amplitude as a function of the cycle iterations. This is an example of scaling on iteration.

Basic algorithm:

Default custom cycle-policy example:

Results:

This result highlights one of the key differences between scaling on cycle vs scaling on iteration. When you scale on cycle, the absolute change in learning rate from one iteration to the next is always constant in a cycle. Scaling on iteration alters the absolute change at every iteration; in this particular case, the absolute change is monotonically decreasing. This results in the curvature between peaks.

During training, you may wish to adjust your cycle parameters:

allows you to start a new cycle w/ new parameters.

also sets the cycle iteration count to zero. If you are using a policy with dynamic amplitude scaling, this ensures the scaling function is reset.

If an argument is not not included in the function call, then the corresponding parameter is unchanged in the new cycle. As a consequence, calling

simply resets the original cycle.

attribute dict. This generated many of the plots above.

Note: iterations in the history is the running training iterations; it is distinct from the cycle iterations and does not reset. This allows you to plot your learning rates over training iterations, even after you change/reset the cycle.

Example:

The author offers a simple approach to determining the boundaries of your cycle by increasing the learning rate over a number of epochs and observing the results. They refer to this as an “LR range test.”

to be the total number of iterations in the number of epochs you wish to test on. This linearly increases the learning rate at each iteration over the range desired.

respectively.

attribute to get the learning rates and accuracy at each iteration.

Note that the clr callback updates the learning rate prior to any further learning rate adjustments as called for in a given optimizer.

clr_callback_tests.ipynb contains tests demonstrating desired behavior of optimizers.