Stemming and lemmatization

Word stemming is a process of reverting an inflected or derived word to its root form. For instance, machine is the stem of machines, and learning and learned are generated from learn as their stem.

The word lemmatization is a cautious version of stemming. It considers the PoS of a word when conducting stemming. We will discuss these two text preprocessing techniques, stemming and lemmatization, in further detail shortly. For now, let's take a quick look at how they're implemented respectively in NLTK by performing the following steps:

1. Import porter as one of the three built-in stemming algorithms (LancasterStemmer and SnowballStemmer are the other two) and initialize the stemmer as follows:

>>> from nltk.stem.porter import PorterStemmer
>>> porter_stemmer = PorterStemmer()

1. Stem machines and learning, as shown in the following codes:

>>> porter_stemmer.stem('machines')
'machin'
>>> porter_stemmer.stem('learning')
'learn'

1. Now import a lemmatization algorithm based on the built-in WordNet corpus and initialize a lemmatizer:

>>> from nltk.stem import WordNetLemmatizer
>>> lemmatizer = WordNetLemmatizer()

Similar to stemming, we lemmatize machines, learning:

>>> lemmatizer.lemmatize('machines')
'machine'
>>> lemmatizer.lemmatize('learning')
'learning'

Why is learning unchanged? It turns out that this algorithm only lemmatizes on nouns by default.