When to stop tuning machine learning models

When to stop tuning the hyperparameters in a machine learning model is a million-dollar question. This problem can be mostly solved by keeping tabs on training and testing errors. While increasing the complexity of a model, the following stages occur:

• Stage 1: Underfitting stage - high train and high test errors (or low train and low test accuracy)
• Stage 2: Good fit stage (ideal scenario) - low train and low test errors (or high train and high test accuracy)
• Stage 3: Overfitting stage - low train and high test errors (or high train and low test accuracy)