"Better Deep Learning" Jason Brownlee Noise topic

Preamble

• This post is an extract of the book "Better Deep Learning" from Jason Brownlee. This post covers how to "Promote Robustness with Noise" described in chapter 17 of the book.

Massif du Brévent

Chapter 17: Promote Robustness with Noise

• Training a neural network with a small dataset can cause the network to memorize all training examples, in turn leading to poor performance on a holdout dataset. Small datasets may also represent a harder mapping problem for neural networks learn, given the patchy or sparse sampling of points in the high -dimensional input space. One approach to making the input space smoother and easier to learn is to add noise to inputs during training.
• "Many studies have noted that adding small amounts of input noise (jitter) to the training data often aids generalization and fault tolerance."
• In effect, adding noise expands the size of the training dataset. Adding noise to input samples is a simple form of data augmentation.
• The most common type of noise used during training is the addition of Gaussian noise to input variables. Gaussian noise, or white noise, has a mean of zero and a standard deviation of one and can be generated as needed using a pseudorandom number generator.
• The addition of Gaussian noise to the inputs to a neural network was traditionally referred to as jitter or random jitter after the use of the term in signal processing to refer to the uncorrelated random noise in electrical circuits.
• Noise is only added during training. No noise is added during the evaluation of the model or when the model is used to make predictions on new data.
• Although additional noise to the inputs is the most common and widely studied approach, random noise can be added to other parts of the network during training.
• Tips for adding noise during training:
• It is appropriate to try adding noise to both classification and regression type problems.
• Add noise to different network types
• Rescale data first
• Test the amount of noise
• Noisy training only 
• Keras supports the addition of noise to models via the GaussianNoise layer.

Noise Regularization Case Study

• We generate a binary classification problem that we are going to solve with a MLP network.

Circles Dataset with Color Showing the Class Value of Each Sample

• Running the MLP network on the binary classification problem:

Cross-entropy loss and Accuracy showing an Overfit

• Running the same MLP but after adding noise to the input values:

Cross-entropy loss and Accuracy with Gaussian noise on the input variables

• Now we run the MLP with noise but the noise is run between the hidden layers:

Cross-entropy loss and Accuracy on a MLP with Gaussian noise between hidden layers

• The last trial is to add the Gaussian noise after the activation function. With this last step, it's the best result in term of accuracy I got:

Cross-entropy loss and Accuracy on a MLP with Gaussian noise injected after the activation function.

Conclusion

• During this chapter 17 of the book "Better Deep Learning", we have been through several code examples on how to introduce Gaussian noise in the model in order to augment the accuracy.
• Adding Gaussian noise to a model improves the accuracy of the model even a model with a low number of observations.