This class converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into their ASCII equivalents, if one exists.

TokenStream from a canned list of Tokens.

Emits no tokens

Deprecated.

Emits the entire input as a single token.

A LetterTokenizer is a tokenizer that divides text at non-letters.

An abstract TokenFilter to make it easier to build graph token filters requiring some lookahead.

LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together.

Randomly inserts overlapped (posInc=0) tokens with posLength sometimes > 1.

Uses LookaheadTokenFilter to randomly peek at future tokens.

TokenFilter that adds random variable-length payloads.

Transforms the token stream as per the Porter stemming algorithm.

This TokenFilter provides the ability to set aside attribute states that have already been analyzed.

A TokenFilter is a TokenStream whose input is another TokenStream.

A TokenStream enumerates the sequence of tokens, either from Fields of a Document or from query text.

A TokenFilter that checks consistency of the tokens (eg offsets are consistent with one another).

Returns true, iff the current state of the passed-in AttributeSource shall be stored in the sink.

Deprecated.

Construct a new LetterTokenizer using a given AttributeSource.

Construct a new LowerCaseTokenizer using a given AttributeSource.

Deprecated.

A TokenStream that uses the same attributes as the supplied one.

Construct a new WhitespaceTokenizer using a given AttributeSource.

Deprecated.

A TokenFilter that applies ArabicStemmer to stem Arabic words..

Deprecated.

A TokenFilter that applies BulgarianStemmer to stem Bulgarian words.

A TokenFilter that applies BrazilianStemmer.

Forms bigrams of CJK terms that are generated from StandardTokenizer or ICUTokenizer.

A TokenFilter that normalizes CJK width differences: Folds fullwidth ASCII variants into the equivalent basic latin Folds halfwidth Katakana variants into the equivalent kana

Deprecated.

Tokenizes input text into sentences.

Base class for decomposition token filters.

A TokenFilter that decomposes compound words found in many Germanic languages.

A TokenFilter that applies CzechStemmer to stem Czech words.

A TokenFilter that applies GermanLightStemmer to stem German words.

Normalizes German characters according to the heuristics of the German2 snowball algorithm.

Normalizes token text to lower case, removes some Greek diacritics, and standardizes final sigma to sigma.

A TokenFilter that applies EnglishMinimalStemmer to stem English words.

A high-performance kstem filter for english.

A TokenFilter that applies SpanishLightStemmer to stem Spanish words.

A TokenFilter that applies PersianNormalizer to normalize the orthography.

A TokenFilter that applies FinnishLightStemmer to stem Finnish words.

Removes elisions from a TokenStream.

A TokenFilter that applies FrenchMinimalStemmer to stem French words.

Normalises token text to lower case, handling t-prothesis and n-eclipsis (i.e., that 'nAthair' should become 'n-athair')

A TokenFilter that applies GalicianMinimalStemmer to stem Galician words.

A TokenFilter that applies HindiNormalizer to normalize the orthography.

A TokenFilter that applies HungarianLightStemmer to stem Hungarian words.

TokenFilter that uses hunspell affix rules and words to stem tokens.

A TokenFilter that applies search term folding to Unicode text, applying foldings from UTR#30 Character Foldings.

A TokenFilter that transforms text with ICU.

Breaks text into words according to UAX #29: Unicode Text Segmentation (http://www.unicode.org/reports/tr29/)

A TokenFilter that applies IndonesianStemmer to stem Indonesian words.

A TokenFilter that applies IndicNormalizer to normalize text in Indian Languages.

A TokenFilter that applies ItalianLightStemmer to stem Italian words.

Replaces term text with the BaseFormAttribute.

Removes tokens that match a set of part-of-speech tags.

Tokenizer for Japanese that uses morphological analysis.

A TokenFilter that applies LatvianStemmer to stem Latvian words.

An always exhausted token stream.

Joins two token streams and leaves the last token of the first stream available to be used when updating the token values in the second stream based on that token.

Provides the ability to override any KeywordAttribute aware stemmer with custom dictionary-based stemming.

Tokenizes the given token into n-grams of given size(s).

Tokenizes the input into n-grams of the given size(s).

Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range

Creates NGramTokenizer with given min and max n-grams.

Deprecated.

A TokenFilter that applies NorwegianLightStemmer to stem Norwegian words.

Tokenizer for path-like hierarchies.

Characters before the delimiter are the "token", those after are the payload.

Adds the Token.setStartOffset(int) and Token.setEndOffset(int) First 4 bytes are the start

TokenFilter for Beider-Morse phonetic encoding.

Create tokens for phonetic matches.

Set the positionIncrement of all tokens to the "positionIncrement", except the first return token which retains its original positionIncrement value.

A TokenFilter that applies PortugueseLightStemmer to stem Portuguese words.

A TokenFilter that applies PortugueseStemmer to stem Portuguese words.

Reverse token string, for example "country" => "yrtnuoc".

Deprecated.

Deprecated.

Deprecated.

A ShingleFilter constructs shingles (token n-grams) from a token stream.

A filter that stems words using a Snowball-generated stemmer.

Normalizes tokens extracted with ClassicTokenizer.

Normalizes tokens extracted with StandardTokenizer.

This class implements Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29 URLs and email addresses are also tokenized according to the relevant RFCs.

Creates a new ClassicTokenizer with a given AttributeSource.

Deprecated.

Transforms the token stream as per the stemming algorithm.

A TokenFilter that applies SwedishLightStemmer to stem Swedish words.

Matches single or multi word synonyms in a token stream.

TokenFilter that use BreakIterator to break each Token that is Thai into separate Token(s) for each Thai word.

Normalizes Turkish token text to lower case.

Extension of StandardTokenizer that is aware of Wikipedia syntax.

Creates a new instance of the WikipediaTokenizer.

Converts each token into its CollationKey, and then encodes the CollationKey with IndexableBinaryStringTools, to allow it to be stored as an index term.

A tokenizer which adds to each category token payload according to the CategoryEnhancements defined in the given EnhancementsIndexingParams.

Tokenizer for associations of a category

An attribute stream built from an Iterable of CategoryAttribute.

This class adds parents to a CategoryAttributesStream.

A base class for all token filters which add term and payload attributes to tokens and are to be used in CategoryDocumentBuilder.

Filter which discards the token 'stop' and which expands the token 'phrase' into 'phrase1 phrase2'

This class is the base of QueryConfigHandler and FieldConfig.

This class can be used to hold any query configuration and no field configuration.

This query configuration handler is used for almost every processor defined in the StandardQueryNodeProcessorPipeline processor pipeline.

This TokenFilter limits the number of tokens while indexing by adding up the current offset.

Performs a clone of all AttributeImpl instances returned in a new AttributeSource instance.

Copies the contents of this AttributeSource to the given target AttributeSource.

An AttributeSource that uses the same attributes as the supplied one.