This class converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into their ASCII equivalents, if one exists.

TokenStream from a canned list of Tokens.

Emits no tokens

Deprecated.

Emits the entire input as a single token.

A LetterTokenizer is a tokenizer that divides text at non-letters.

An abstract TokenFilter to make it easier to build graph token filters requiring some lookahead.

LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together.

Randomly inserts overlapped (posInc=0) tokens with posLength sometimes > 1.

Uses LookaheadTokenFilter to randomly peek at future tokens.

TokenFilter that adds random variable-length payloads.

Transforms the token stream as per the Porter stemming algorithm.

This TokenFilter provides the ability to set aside attribute states that have already been analyzed.

A TokenFilter is a TokenStream whose input is another TokenStream.

Removes tokens whose types appear in a set of blocked types from a token stream.

A WhitespaceTokenizer is a tokenizer that divides text at whitespace.

The source of tokens for this filter.

Returns the sink TokenStream

Creates a TokenStream which tokenizes all the text in the provided Reader.

This method uses ReusableAnalyzerBase.createComponents(String, Reader) to obtain an instance of ReusableAnalyzerBase.TokenStreamComponents and returns the sink of the components.

Create a new KeywordMarkerFilter, that marks the current token as a keyword if the tokens term buffer is contained in the given set via the KeywordAttribute.

Build a filter that removes words that are too long or too short from the text.

Build a filter that only accepts tokens up to a maximum number.

Deprecated.

Deprecated.

Constructs a filter which removes words from the input TokenStream that are named in the Set.

Instantiates a new TeeSinkTokenFilter.

Creates a new ReusableAnalyzerBase.TokenStreamComponents instance.

The name arg is used to identify this stage when throwing exceptions (useful if you have more than one instance in your chain).

Deprecated.

A TokenFilter that applies ArabicStemmer to stem Arabic words..

A TokenFilter that applies BulgarianStemmer to stem Bulgarian words.

A TokenFilter that applies BrazilianStemmer.

Creates a new BrazilianStemFilter

Forms bigrams of CJK terms that are generated from StandardTokenizer or ICUTokenizer.

A TokenFilter that normalizes CJK width differences: Folds fullwidth ASCII variants into the equivalent basic latin Folds halfwidth Katakana variants into the equivalent kana

Calls CJKBigramFilter(HAN | HIRAGANA | KATAKANA | HANGUL)

Deprecated.

Tokenizes input text into sentences.

Construct a new WordTokenizer.

Base class for decomposition token filters.

A TokenFilter that decomposes compound words found in many Germanic languages.

Deprecated.

Deprecated.

Deprecated.

Deprecated.

Deprecated.

Deprecated.

Creates a new DictionaryCompoundWordTokenFilter

Deprecated.

Deprecated.

Create a HyphenationCompoundWordTokenFilter with no dictionary.

Deprecated.

Creates a new HyphenationCompoundWordTokenFilter instance.

A TokenFilter that applies CzechStemmer to stem Czech words.

A TokenFilter that applies GermanLightStemmer to stem German words.

Normalizes German characters according to the heuristics of the German2 snowball algorithm.

Deprecated.

Normalizes token text to lower case, removes some Greek diacritics, and standardizes final sigma to sigma.

Deprecated.

A TokenFilter that applies EnglishMinimalStemmer to stem English words.

A high-performance kstem filter for english.

A TokenFilter that applies SpanishLightStemmer to stem Spanish words.

A TokenFilter that applies PersianNormalizer to normalize the orthography.

A TokenFilter that applies FinnishLightStemmer to stem Finnish words.

Removes elisions from a TokenStream.

A TokenFilter that applies FrenchMinimalStemmer to stem French words.

Deprecated.

Deprecated.

Constructs an elision filter with a Set of stop words

Deprecated.

Normalises token text to lower case, handling t-prothesis and n-eclipsis (i.e., that 'nAthair' should become 'n-athair')

Create an IrishLowerCaseFilter that normalises Irish token text.

A TokenFilter that applies GalicianMinimalStemmer to stem Galician words.

A TokenFilter that applies HindiNormalizer to normalize the orthography.

A TokenFilter that applies HungarianLightStemmer to stem Hungarian words.

TokenFilter that uses hunspell affix rules and words to stem tokens.

Creates a new HunspellStemFilter that will stem tokens from the given TokenStream using affix rules in the provided HunspellDictionary

A TokenFilter that applies search term folding to Unicode text, applying foldings from UTR#30 Character Foldings.

A TokenFilter that transforms text with ICU.

Create a new ICUFoldingFilter on the specified input

Create a new Normalizer2Filter with the specified Normalizer2

Breaks text into words according to UAX #29: Unicode Text Segmentation (http://www.unicode.org/reports/tr29/)

A TokenFilter that applies IndonesianStemmer to stem Indonesian words.

Calls IndonesianStemFilter(input, true)

A TokenFilter that applies IndicNormalizer to normalize text in Indian Languages.

A TokenFilter that applies ItalianLightStemmer to stem Italian words.

Replaces term text with the BaseFormAttribute.

Removes tokens that match a set of part-of-speech tags.

Tokenizer for Japanese that uses morphological analysis.

A TokenFilter that applies LatvianStemmer to stem Latvian words.

An always exhausted token stream.

Joins two token streams and leaves the last token of the first stream available to be used when updating the token values in the second stream based on that token.

Provides the ability to override any KeywordAttribute aware stemmer with custom dictionary-based stemming.

Create a new StemmerOverrideFilter, performing dictionary-based stemming with the provided dictionary.

Tokenizes the given token into n-grams of given size(s).

Tokenizes the input into n-grams of the given size(s).

Creates EdgeNGramTokenFilter that can generate n-grams in the sizes of the given range

Creates NGramTokenFilter with default min and max n-grams.

Deprecated.

Deprecated.

A TokenFilter that applies NorwegianLightStemmer to stem Norwegian words.

Tokenizer for path-like hierarchies.

Characters before the delimiter are the "token", those after are the payload.

Adds the Token.setStartOffset(int) and Token.setEndOffset(int) First 4 bytes are the start

TokenFilter for Beider-Morse phonetic encoding.

Create tokens for phonetic matches.

Calls BeiderMorseFilter(input, engine, null)

Set the positionIncrement of all tokens to the "positionIncrement", except the first return token which retains its original positionIncrement value.

Constructs a PositionFilter that assigns a position increment of zero to all but the first token from the given input stream.

A TokenFilter that applies PortugueseLightStemmer to stem Portuguese words.

A TokenFilter that applies PortugueseStemmer to stem Portuguese words.

Reverse token string, for example "country" => "yrtnuoc".

Deprecated.

Create a new ReverseStringFilter that reverses all tokens in the supplied TokenStream.

Deprecated.

Deprecated.

A ShingleFilter constructs shingles (token n-grams) from a token stream.

Construct a ShingleFilter with default shingle size: 2.

Constructs a ShingleFilter with the specified shingle size from the TokenStream input

Creates a shingle filter using default settings.

Creates a shingle filter using the default ShingleMatrixFilter.TokenSettingsCodec.

A filter that stems words using a Snowball-generated stemmer.

Returns a (possibly reused) StandardTokenizer filtered by a StandardFilter, a LowerCaseFilter, a StopFilter, and a SnowballFilter

Construct the named stemming filter.

Normalizes tokens extracted with ClassicTokenizer.

Normalizes tokens extracted with StandardTokenizer.

This class implements Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29 URLs and email addresses are also tokenized according to the relevant RFCs.

Construct filtering in.

Transforms the token stream as per the stemming algorithm.

Create filter using the supplied stemming table.

A TokenFilter that applies SwedishLightStemmer to stem Swedish words.

Matches single or multi word synonyms in a token stream.

TokenFilter that use BreakIterator to break each Token that is Thai into separate Token(s) for each Thai word.

Deprecated.

Normalizes Turkish token text to lower case.

Create a new TurkishLowerCaseFilter, that normalizes Turkish token text to lower case.

Extension of StandardTokenizer that is aware of Wikipedia syntax.

Converts each token into its CollationKey, and then encodes the CollationKey with IndexableBinaryStringTools, to allow it to be stored as an index term.

The TokesStream for this field to be used when indexing, or null.

Returns a NumericTokenStream for indexing the numeric value.

Expert: sets the token stream to be used for indexing and causes isIndexed() and isTokenized() to return true.

Create a tokenized and indexed field that is not stored.

A tokenizer which adds to each category token payload according to the CategoryEnhancements defined in the given EnhancementsIndexingParams.

Get the CategoryListTokenizer which generates the category list for this enhancement.

Constructor.

Tokenizer for associations of a category

Get a stream of categories which includes the parents, according to policies defined in indexing parameters.

Get a category list tokenizer (or a series of such tokenizers) to create the category list tokens.

Get a CountingListTokenizer for creating counting list token.

An attribute stream built from an Iterable of CategoryAttribute.

This class adds parents to a CategoryAttributesStream.

A base class for all token filters which add term and payload attributes to tokens and are to be used in CategoryDocumentBuilder.

Constructor.

Convenience method; Creates and returns a token stream that generates a token for each keyword in the given collection, "as is", without any transforming text analysis.

Equivalent to addField(fieldName, stream, 1.0f).

Filter which discards the token 'stop' and which expands the token 'phrase' into 'phrase1 phrase2'

This TokenFilter limits the number of tokens while indexing by adding up the current offset.

A convenience method that tries a number of approaches to getting a token stream.

Highlights chosen terms in a text, extracting the most relevant section.

Highlights terms in the text , extracting the most relevant sections and concatenating the chosen fragments with a separator (typically "...").

Creates a Map of WeightedSpanTerms from the given Query and TokenStream.

Creates a Map of WeightedSpanTerms from the given Query and TokenStream.

Initializes the Fragmenter.