Feb 15, 2012
Solr Tokenizer
Tokenizer is an analyzer that takes text and splits it into smaller units.
Eg:
"Hello World" => "Hello","World" (Tokenized on whitespace, generated 2 tokens)
Some of the commonly used tokenizer are mentioned below:
WhitespaceTokenizerFactory: Tokenizes on whitespace. Above is the example of this factory.
KeywordTokenizerFactory: This doesn't do any tokenization but returns as it is. Is is needed when you want to perform exact match.
StandardTokenizerFactory: This analyzer has the most practical use. It tokenizes on whitespace, special characters etc. Please refer the apache site for detailed description.
There are several other tokenizers like LetterTokenizerFactory, HTMLStripWhitespaceTokenizerFactory etc.