"Better Deep Learning" Jason Brownlee Early stopping topic

Preamble

• This post is an excerpt from Chapter 18 of the book "Better Deep Learning" from Jason Brownlee.
• The series of post related to Deep Learning are here to help me in the future when working on Machine Learning so that it's easy to get back and refreshed on specific topics such as this one.
• Each post is written in parallel with the reading of the book. I also report in this post the different case study provided in the book.

Téléphérique Aiguille du Midi

Chapter 18: Halt Training at the Right Time with Early Stopping

• A major challenge in trading neural networks is how long to train them. Too little training will mean that the model will underfit the train and the test sets. Too much training will mean that the model will overfit the training dataset and have poor performance on the test set. A compromise is to train on the training dataset but to stop training at the point where performance on a validation dataset starts to degrade. This simple, effective, and widely used approach to training neural networks is called early stopping.
• It is common to split the training dataset and use a subset, such as 30%, as a validation dataset used to monitor performance of the model during the training. This validation set is not used to train the model. It is also common to use the loss on a validation dataset as the metric to monitor, although you may also use prediction error in the case of regression, or accuracy in the case of classification.
• At the time that training is halted, the model is known to have slightly worse generalization error than a model at a prior epoch.
• Perhaps a simple approach is to always save the model weights if the performance of the model on a holdout dataset is better than at the previous epoch.
• Early stopping could be used with k-fold cross-validation, although it is not recommended.
• A callback is a snippet of code that can be executed at a specific point during training, such as before or after training, an epoch or a batch. They provide a way to execute code and interact with the training model process automatically. Callbacks can be provided to the fit() function via the callbacks argument.
• Keras supports the early stopping of training via a callback called EarlyStopping.
• Saving and loading models requires that HDF5 support has been installed on your workstation.

Early Stopping Case Study

• We are using a classical binary classification problem as guinea pig to test early stopping on MLP:

A binary classification problem with two classes

• The first step of the case study is to establish a baseline. We run a classical MLP without any means to address the overfitting:

MLP with Overfitting

• As for the second step, we introduce early stopping but the performance result is worse:

MLP on classification problem with early Stopping

• So as the third step, we try to add a new parameter which is "patience"=200, which means that we will wait 200 epochs more before training is stopped:

Plot of Cross-Entropy Loss and Accuracy of a MLP with Early Stopping and patience

• As a final step we can try to search what is the best moment to stop the training by using a ModelCheckkpoint callback. In this case we are interested in saving the model with the best accuracy on the dataset:

Plot of Cross-Entropy and Accuracy of a MLP with Early Stopping and Checkpointing

Conclusion

• We were able to test the ability to early stop a model evaluation in order to address the overfitting problem.
• Up to Keras API and callback, we are able to monitor the performance of a MLP network so that we can stop the training of the model before the model overfits.
• Thanks to Jason Brownlee for this excellent book. As always the code examples are perfect without any bug.