Building Your NLP Vocabulary

In the earlier chapters, you were introduced to why Natural Language Processing (NLP) is important especially in today's context, which was followed by a discussion on a few prerequisites and Python libraries that are highly beneficial for NLP tasks. In this chapter, we will take this discussion further and discuss some of the most concrete tasks involved in building a vocabulary for NLP tasks and preprocessing textual data in detail. We will start by learning what a vocabulary is and take the notion forward to actually build a vocabulary. We will do this by applying various methods on text data that are present in most of the NLP pipelines across any organization.

In this chapter, we'll cover the following topics:

• Lexicons
• Phonemes, graphemes, and morphemes
• Tokenization
• Understanding word normalization