Defining custom analyzers

It's necessary to create a custom analyzer when the built-in analyzers do not provide the needed behaviors for your search application. To continue with our CourtesyTitleFilter example, we will create CourtesyTitleAnalyzer.

The anatomy of an analyzer includes one tokenizer and one or more TokenFilters. We will build an Analyzer by extending from the Analyzer abstract class and implement the createComponents method.

How to do it…

Here is the sample code for CourtesyTitleAnalyzer:

public class CourtesyTitleAnalyzer extends Analyzer {

    @Override
    protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
    Tokenizer letterTokenizer = new LetterTokenizer(reader);
    TokenStream filter = new CourtesyTitleFilter(letterTokenizer);
    return new TokenStreamComponents(letterTokenizer, filter);
  }
}

How it works…

An Analyzer is created by extending from the Analyzer abstract class as shown in this example. Then we override the createComponents method, adding a LetterTokenizer to split text by non-letter characters and CourtesyTitleFilter as a TokenFilter. Finally, we return a new TokenStreamComponents instance initialized by the instantiated Tokenizer and TokenFilter.

Note that the only method we need to override is createComponents. We don't need to override the constructor to build our Analyzer because components are not added during construction; they are added when the createComponents method is called. Therefore, we override the createComponents method to customize an Analyzer. Also note that we cannot override the tokenStream method because it's declared as final.