Prediction using linear regression

Aaron, a friend of mine, is a little sloppy with money and is never able to estimate how much his monthly credit card bill will be. Can we do something to help him? Well, yes, linear regression can help us to predict a monthly credit card bill if we have sufficient data. Thanks to the digital economy, all of his monetary transactions for the last five years are available online. We extracted his monthly expenditure on groceries, stationery, and travel and his monthly income. Linear regression helped not only in predicting his monthly credit card bill, it also gave an insight into which factor was most responsible for his spending.

This was just one example; linear regression can be used in many similar tasks. In this section, we'll learn how we can perform linear regression on our data.

Linear regression is a supervised learning task. It's one of the most basic, simple, and extensively used ML techniques for prediction. The goal of regression is to find a function F(x, W), for a given input-output pair (x, y), so that y = F(x, W). In the (x, y) pair, x is the independent variable and y the dependent variable, and both of them are continuous variables. It helps us to find the relationship between the dependent variable y and the independent variable(s) x. 

The input x can be a single input variable or many input variables. When F(x, W) maps a single input variable x, it's called simple linear regression; for multiple input variables, it's called multiple linear regression. 

The function F(x, W) is approximated using the following expression:

In this expression, d is the dimensions of x (number of independent variables), and W is the weight associated with each component of x. To find the function F(x, W), we need to determine the weights. The natural choice is to find the weights that reduce the squared error, hence our objective function is as follows:

In the preceding function, N is the total number of the input-output pair presented. To find the weights, we differentiate the objective function with respect to weight and equate it to 0. In matrix notation, we can write the solution for the column vector W = (W₀, W₁, W₂, ..., W_d)T as follows:

On differentiating and simplifying, we get the following:

X is the input vector of size [N, d] and Y the output vector of size [N, 1]. The weights can be found if (XTX)-1 exists, that's if all of the rows and columns of X are linearly independent. To ensure this, the number of input-output samples (N) should be much greater than the number of input features (d).