Comparing underfitting and overfitting

In the preceding list, step 4 implies an iterative process where we try models, parameters, and features until we get the best result that we can. Let's now think about a classification problem, where we want to separate squares from circles, as shown in the following diagram. At the beginning of the process, we will probably be in a situation that is similar to the first chart (on the left-hand side). The model fails to efficiently separate the two shapes and both sides are a mixture of both squares and circles. This is called underfitting and refers to a model that fails to represent the characteristics of the dataset:

As we continue tuning parameters and adjusting the model to the training dataset, we might find ourselves in a situation that is similar to the third chart (on the right-hand side). The model accurately splits the dataset, leaving only one shape on each side of the border line. Even if this seems correct, it completely lacks generalization. The result adjusts so well to the training data that it will be completely wrong to we test it against a different dataset. This problem is called overfitting.

To solve the problem of overfitting in our model, we need to increase its adaptability. However, making it too flexible can also make it bad at predicting. To avoid this, the usual solution is to use regularization techniques. There are many similar techniques that can be found in specialized literature, but they are beyond the scope of this book.

The center chart shows a more flexible model; it represents the dataset, but is general enough to deal with new, previously unseen data. It is often time-consuming and it can be difficult to get the right balance in order to build a good machine learning model.