This class converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into their ASCII equivalents, if one exists.

Abstract base class for TokenFilters that may remove tokens.

Marks terms as keywords via the KeywordAttribute.

This TokenFilter limits the number of tokens while indexing.

Normalizes token text to lower case.

Randomly inserts overlapped (posInc=0) tokens with posLength sometimes > 1.

Uses LookaheadTokenFilter to randomly peek at future tokens.

Transforms the token stream as per the Porter stemming algorithm.

This TokenFilter provides the ability to set aside attribute states that have already been analyzed.

A TokenFilter that checks consistency of the tokens (eg offsets are consistent with one another).

A TokenFilter that applies ArabicNormalizer to normalize the orthography.

A TokenFilter that applies BulgarianStemmer to stem Bulgarian words.

A TokenFilter that applies BrazilianStemmer.

Forms bigrams of CJK terms that are generated from StandardTokenizer or ICUTokenizer.

Deprecated.

A TokenFilter that breaks sentences into words.

Base class for decomposition token filters.

A TokenFilter that decomposes compound words found in many Germanic languages.

A TokenFilter that applies CzechStemmer to stem Czech words.

A TokenFilter that applies GermanLightStemmer to stem German words.

Normalizes German characters according to the heuristics of the German2 snowball algorithm.

Normalizes token text to lower case, removes some Greek diacritics, and standardizes final sigma to sigma.

A TokenFilter that applies EnglishMinimalStemmer to stem English words.

A high-performance kstem filter for english.

A TokenFilter that applies SpanishLightStemmer to stem Spanish words.

A TokenFilter that applies PersianNormalizer to normalize the orthography.

A TokenFilter that applies FinnishLightStemmer to stem Finnish words.

Removes elisions from a TokenStream.

A TokenFilter that applies FrenchMinimalStemmer to stem French words.

Normalises token text to lower case, handling t-prothesis and n-eclipsis (i.e., that 'nAthair' should become 'n-athair')

A TokenFilter that applies GalicianMinimalStemmer to stem Galician words.

A TokenFilter that applies HindiNormalizer to normalize the orthography.

A TokenFilter that applies HungarianLightStemmer to stem Hungarian words.

TokenFilter that uses hunspell affix rules and words to stem tokens.

A TokenFilter that applies search term folding to Unicode text, applying foldings from UTR#30 Character Foldings.

A TokenFilter that transforms text with ICU.

A TokenFilter that applies IndonesianStemmer to stem Indonesian words.

A TokenFilter that applies IndicNormalizer to normalize text in Indian Languages.

A TokenFilter that applies ItalianLightStemmer to stem Italian words.

Replaces term text with the BaseFormAttribute.

Removes tokens that match a set of part-of-speech tags.

A TokenFilter that applies LatvianStemmer to stem Latvian words.

Provides the ability to override any KeywordAttribute aware stemmer with custom dictionary-based stemming.

Tokenizes the given token into n-grams of given size(s).

Deprecated.

A TokenFilter that applies NorwegianLightStemmer to stem Norwegian words.

Characters before the delimiter are the "token", those after are the payload.

Adds the Token.setStartOffset(int) and Token.setEndOffset(int) First 4 bytes are the start

TokenFilter for Beider-Morse phonetic encoding.

Create tokens for phonetic matches.

Set the positionIncrement of all tokens to the "positionIncrement", except the first return token which retains its original positionIncrement value.

A TokenFilter that applies PortugueseLightStemmer to stem Portuguese words.

A TokenFilter that applies PortugueseStemmer to stem Portuguese words.

Reverse token string, for example "country" => "yrtnuoc".

A TokenFilter that applies RussianLightStemmer to stem Russian words.

Deprecated.

A ShingleFilter constructs shingles (token n-grams) from a token stream.

A filter that stems words using a Snowball-generated stemmer.

Normalizes tokens extracted with ClassicTokenizer.

Transforms the token stream as per the stemming algorithm.

A TokenFilter that applies SwedishLightStemmer to stem Swedish words.

Matches single or multi word synonyms in a token stream.

TokenFilter that use BreakIterator to break each Token that is Thai into separate Token(s) for each Thai word.

Normalizes Turkish token text to lower case.

Converts each token into its CollationKey, and then encodes the CollationKey with IndexableBinaryStringTools, to allow it to be stored as an index term.

A tokenizer which adds to each category token payload according to the CategoryEnhancements defined in the given EnhancementsIndexingParams.

Tokenizer for associations of a category

A base class for category list tokenizers, which add category list tokens to category streams.

Basic class for setting the CharTermAttributes and PayloadAttributes of category tokens.

CategoryListTokenizer for facet counting

Filter which discards the token 'stop' and which expands the token 'phrase' into 'phrase1 phrase2'

This TokenFilter limits the number of tokens while indexing by adding up the current offset.