This class converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into their ASCII equivalents, if one exists.

An abstract base class for simple, character-oriented tokenizers.

Deprecated.

Emits the entire input as a single token.

A LetterTokenizer is a tokenizer that divides text at non-letters.

Normalizes token text to lower case.

Expert: This class provides a TokenStream for indexing numeric values that can be used by NumericRangeQuery or NumericRangeFilter.

Removes stop words from a token stream.

TokenStream output from a tee with optional filtering.

A Tokenizer is a TokenStream whose input is a Reader.

A WhitespaceTokenizer is a tokenizer that divides text at whitespace.

The source of tokens for this filter.

Returns the sink TokenStream

This method uses ReusableAnalyzerBase.createComponents(String, Reader) to obtain an instance of ReusableAnalyzerBase.TokenStreamComponents.

This method uses ReusableAnalyzerBase.createComponents(String, Reader) to obtain an instance of ReusableAnalyzerBase.TokenStreamComponents and returns the sink of the components.

Create a new KeywordMarkerFilter, that marks the current token as a keyword if the tokens term buffer is contained in the given set via the KeywordAttribute.

Build a filter that removes words that are too long or too short from the text.

Build a filter that only accepts tokens up to a maximum number.

Create a new LowerCaseFilter, that normalizes token text to lower case.

Deprecated.

Constructs a filter which removes words from the input TokenStream that are named in the Set.

Instantiates a new TeeSinkTokenFilter.

Creates a new ReusableAnalyzerBase.TokenStreamComponents instance.

Normalizes tokens extracted with ClassicTokenizer.

Normalizes tokens extracted with StandardTokenizer.

This class implements Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29 URLs and email addresses are also tokenized according to the relevant RFCs.

Construct filtering in.

Converts each token into its CollationKey, and then encodes the CollationKey with IndexableBinaryStringTools, to allow it to be stored as an index term.

The TokesStream for this field to be used when indexing, or null.

Returns a NumericTokenStream for indexing the numeric value.

Expert: sets the token stream to be used for indexing and causes isIndexed() and isTokenized() to return true.

Create a tokenized and indexed field that is not stored.