Regularization overview

You may recall that our linear model follows the form: Y = B0 + B₁x₁ +...B_nx_n + e, and that the best fit tries to minimize the RSS, which is the sum of the squared errors of the actual minus the estimate, or e₁2 + e₂2 + ... e_n2.

With regularization, we'll apply what is known as a shrinkage penalty in conjunction with RSS minimization. This penalty consists of a lambda (symbol λ), along with the normalization of the beta coefficients and weights. How these weights are normalized differs in terms of techniques, and we'll discuss them accordingly. Quite simply, in our model, we're minimizing (RSS + λ (normalized coefficients)). We'll select λ, which is known as the tuning parameter, in our model building process. Please note that if lambda is equal to 0, then our model is equivalent to OLS, as it cancels out the normalization term. As we work through this chapter, the methods can be applied to a classification problem.

So what does regularization do for us and why does it work? First of all, regularization methods are very computationally efficient. In a best subsets of features, we're searching 2p models and, in large datasets, it isn't feasible to attempt this. In the techniques that follow, we only fit one model to each value of lambda and, as you can imagine this, is far less computationally demanding. Another reason goes back to our bias-variance trade-off, discussed in the preface. In the linear model, where the relationship between the response and the predictors is close to linear, the least squares estimates will have low bias but may have high variance. This means that a small change in the training data can cause a significant change in the least squares coefficient estimates (James, 2013). Regularization through the proper selection of lambda and normalization may help you improve the model fit by optimizing the bias-variance trade-off. Finally, the regularization of coefficients may work to solve multicollinearity problems as we shall see.