Better Deep Learning - Jason Brownlee - Resampling Ensembles

 

Preamble

• This post is an extract of the book "Better Deep Learning" from Jason Brownlee. 
• This post is related to "Better Predictions": how to make better predictions using Resampling Ensembles.
• Best sentences from the book are extracted as a reminder for me in the future.
• I run all code examples delivered with the book on my iMac environment (Anaconda + Spyder) and the results of my experimentations are reported in this post.

Eglise Sofia - Bulgarie

Chapter 22: Fit Models on Different Samples with Resampling Ensembles

• One way to achieve differences between models is to train each model on a different subset of the available training data. Models are trained on different subsets of the training data naturally through the use of resampling methods such as cross-validation and the bootstrap, designed to estimate the average performance of the model generally on unseen data. The models used in this estimation process can be combined in what is referred to as a resampling-based ensemble, such as a cross-validation ensemble or a bootstrap aggregation (or bagging) ensemble.
• Thanks to the book of Jason Brownlee, this chapter will help us to know more about how to estimate model performance using random-splits and develop an ensemble from the models. 
• We will see how to estimate performance using 10-fold cross-validation and develop a cross-validation ensemble. 
• And finally we will see how to estimate performance using then bootstrap and combine models using a bagging ensemble.
• Effective ensembles require members that disagree. Each member must have skill (e.g. perform better than random chance), but ideally, perform well in different ways. Technically, we can say that we prefer ensemble members to have low correlation in their predictions, or prediction errors.
• Multiple models are fit using slightly different perspectives on the training data and, in turn, make different errors and often more stable and better predictions when combined. We can refer to these methods generally as data resampling ensembles. A benefit of this approach is that resampling methods may be used that do not make use of all examples in the training data set. Any examples that are not used to fit the model can be used as a test dataset to estimate the generalization error of the chosen model configuration. There are three popular methods that we could use to create a resampling ensemble; they are:
• Random splits: the dataset is repeatedly sampled with a random split of the data into train and test sets.
• k-fold Cross-Validation: the dataset is split into k equally sized folds, k models are trained and each fold is given an opportunity to be used as the holdout set where the model is trained on all remaining folds.
• Boostrap Aggregation: random samples are collected with replacement and examples not included in a given sample are used as the test set.

Case Study

• The case is how to use the resampling ensemble to reduce the variance of an MLP on a simple multi class problem:

A multi class Classification problem to be submitted to a neural network

• We define a simple Multilayer Perceptron Model in oder to learn the problem. The model will predict a vector with three elements with the probability that the sample belongs to each of the three classes.
• 90% of the data is used for training and 10% of data for the test set. The author explains that it is because it is noisy problem and a well-performing model requires as much data as possible to learn the complex classification problem.

MLP Model Accuracy on Train and Test Dataset without Random Splits Ensemble

• The next step consists of using the technique of random splits ensemble. For that purpose, the author suggests to combine multiple models trained on the random splits with the expectation that performance of the ensemble is likely to be more stable and better than the single average model.
• The author suggests to generate 10 times more sample points from the problem domain and hold them back as an unseen dataset.

Random Splits Ensemble Performance on the classification problem

• As a next step, we try the technique of Cross-Validation Ensemble. This approach is designed to be less optimistic. 
• The procedure has a single parameter called k that refers to the number of groups that a given data sample is to be split into. A typical value for k is 10.
• KFold class from scikit-learn can split the dataset into k folds.
• 
  

Accuracy of a MLP Multiclass Classification problem with Cross-Validation Resampling

• As a final step, we experiment the Bagging Ensemble.
• A limitation of random splits and k-fold cross-validation from the perspective of ensemble learning is that the models are very similar. The bootstrap method is a statistical technique for estimating quantities about a population by averaging estimates from multiple small data samples. Importantly, samples are constructed by drawing observations from a large data sample one at a time and returning them to the data sample after they have been chosen. This allows a given observation to be included in a given call sample more than once. This approach to sampling is called sampling with replacement.
• The bootstrap is a robust method for estimating model performance. It does suffer a little from an optimistic bias, but is often almost as accurate as k-fold cross-validation in practice.
• Generally, use of the bootstrap method in ensemble learning is referred to as bootstrap aggregation or bagging.

Accuracy of a MLP Multiclass Classification problem with Bagging Ensemble

Conclusion

• During this journey across ensembles, we experiment different kind of ensembles:
• Random Split Ensembles
• Cross-validation Ensemble
• Bagging Ensemble (or Bootstrap Aggregating)
• Big thanks to Jason Brownlee for sharing his expertise in this field of deep learning.