Better Deep Learning Jason Brownlee Weighted Average Ensemble

 Preamble

• This post is an extract of the book "Better Deep Learning" from Jason Brownlee. 
• This post is related to "Better Predictions": how to make better predictions using Weighted Average Ensembles.

Lisbonne

Chapter 21: Contribute Proportional to Trust with Weighted Average Ensemble

• It is well known that a combination of many different predictions can improve prediction.
• Learning of continuous valued function using neural network ensembles (committees) can improve accuracy, reliable estimation of the generalization error and active learning.
• Most often the networks in the ensemble are trained individually and their prediction are combined. This combination is usually done by majority (in classification) or by simple averaging (in regression), but one can also use a weighted combination of the networks.
• A modeling averaging ensemble combines the prediction from each model equally and often results in better performance on average than a given single model. Sometimes there are very good models that we wish to contribute more to an ensemble prediction, and perhaps less skillful models that may be useful but should contribute less to an ensemble prediction. A weighted average ensemble is an approach that allows multiple models to contribute to a prediction in proportion to their trust or estimated performance.
• A weighted ensemble is an extension of a model averaging ensemble where the contribution of each member to the final prediction is weighted by the performance of the model.

Case Study

• A small multi class classification problem will be used to demonstrate the weighted averaging ensemble.

Three Classes 

• The problem is a multi class classification problem, and we will model it using a softmax activation function on the output layer. This means that the model will predict a vector with three elements with the probability that the sample belongs to each of the three classes.

MLP without Averaging Ensemble 

• As a first step, we develop a simple model averaging ensemble before we look at developing a weighted average ensemble:

Single Model Accuracy (blue dots) and Accuracy of Ensembles of Increasing Size (orange line)

• Now we can extend, in a second step, with a weighted model of ensemble:
• The model averaging ensemble allows each ensemble member to contribute an equal amount to the prediction of the ensemble. In this second step experimentation, the contribution of each ensemble member is weighted by a coefficient that indicates the trust or expected performance of the model. 
• We can use efficient NumPy functions to implement the weighted sum such as einsum() or tensor dot().
•  The experimentation consists of finding the best combination of weighted ensemble:

Performance of a Weighted Average MLP Ensemble

• As a final step, we will use a directed optimization process, with the help of the SciPy library which provides the differential_evolution() function:

Performance of a Weighted Average MLP Ensemble

• As a final note of these experimentations, I notice that the weighted average model out-perform the model averaging ensemble and individual models.

Conclusion

• During this chapter, we were able, thanks to the explanation and code provided by Jason Brownlee, to evaluate  the model averaging ensemble and the weighted average ensembles.
• The model averaging ensembles are limited because each of them are contributing equally to the global prediction.
• The weighted average ensembles provide a way for each model to contribute to the prediction with a weight that is proportional to the trust or performance of the model.