Feature selection

The number of explanatory features (input variables) of a sample can be enormous wherein you get x_i=(x_i¹, x_i², x_i³, ... , x_i^d) as a training sample (observation/example) and d is very large. An example of this can be a document classification task3, where you get 10,000 different words and the input variables will be the number of occurrences of different words.

This enormous number of input variables can be problematic and sometimes a curse because we have many input variables and few training samples to help us in the learning procedure. To avoid this curse of having an enormous number of input variables (curse of dimensionality), data scientists use dimensionality reduction techniques in order to select a subset from the input variables. For example, in the text classification task they can do the following:

• Extracting relevant inputs (for instance, mutual information measure)
• Principal component analysis (PCA)
• Grouping (cluster) similar words (this uses a similarity measure)