Understanding the scenario of overfitting

In some of the previous recipes, we have noticed that the training accuracy is ~100%, while test accuracy is ~98%, which is a case of overfitting on top of a training dataset. Let's gain an intuition of the delta between the training and the test accuracies.

To understand the phenomenon resulting in overfitting, let's contrast two scenarios where we compare the training and test accuracies along with a histogram of the weights:

• Model is run for five epochs
• Model is run for 100 epochs

The comparison-of-accuracy metric between training and test datasets between the two scenarios is as follows:

Once we plot the histogram of weights that are connecting the hidden layer to the output layer, we will notice that the 100-epochs scenario has a higher spread of weights when compared to the five-epochs scenario:

From the preceding pictures, you should note that the 100 epochs scenario had a higher dispersion of weight values when compared to the five-epochs scenario. This is because of the higher amount of opportunity that the model had to overfit on top of the training dataset when the model is run for 100-epochs, when compared to when the model is run for five epochs, as the number of weight updates in the 100-epochs scenario is higher than the number of weight updates in the five-epochs scenario.

A high value of weight (along with a difference in the training and test dataset) is a good indication of a potential over-fitting of the model and/or a potential opportunity to scale input/weights to increase the accuracy of the model.

Additionally, also note that a neural network can have hundreds of thousands of weights (and millions in certain architectures) that need to be adjusted, and thus, there is always a chance that one or the other weight can get updated to a very high number to fine-tune for one outlier row of the dataset.