Measuring the deep neural network performance

Is a deep network really better than an MLP on this problem? Let's find out! After training for 500 epochs, here's how the model performed:

Model Train MAE: 0.0753991873787
Model Val MAE: 0.189703853999
Model Test MAE: 0.190189985043

We can see that the Train MAE has now decreased from 0.19 to 0.075. We've greatly reduced the bias of the network. 

However, our variance has increased. The difference between the training error and validation error is much larger. Our Val set error did move down slightly, which is good; however, this large gap between training error and validation error suggests we are starting to over fit on the training set.

The most straightforward way to reduce variance in cases like this is to either add additional training data or apply a regularization technique such as L2 regularization or dropout, which we will cover in the next chapter.

Once a network has been built, I like to inspect the errors visually to get a feel for how well the network is modelling the validation set distribution. This often leads to insights that will help me improve the model. For a regression model, I like to plot a histogram of the predicted and actual values of the validation set. Let's see how well I did. The plot is as follows for your reference:

>

Overall, I think the model is predicting the actual distribution fairly closely. It appears that the actual validation dataset is shifted slightly more to the left (smaller values) than the predicted dataset, which may be an important insight. Said another way, the network may be predicting wines have more alcohol than they do on average, especially in the cases where alcohol is somewhat low. Examining the validation data more closely might suggest how we would go about collecting more training data.