Package org.apache.lucene.analysis.ar

Class ArabicAnalyzer

  • All Implemented Interfaces:
    Closeable, AutoCloseable
    public final class ArabicAnalyzer
extends org.apache.lucene.analysis.StopwordAnalyzerBase
    Analyzer for Arabic.

    This analyzer implements light-stemming as specified by: Light Stemming for Arabic Information Retrieval http://www.mtholyoke.edu/~lballest/Pubs/arab_stem05.pdf

    The analysis package contains three primary components:

    • Nested Class Summary

      • Nested classes/interfaces inherited from class org.apache.lucene.analysis.ReusableAnalyzerBase

        org.apache.lucene.analysis.ReusableAnalyzerBase.TokenStreamComponents

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static String DEFAULT_STOPWORD_FILE
      File containing default Arabic stopwords.
      static String STOPWORDS_COMMENT
      Deprecated.
      use WordlistLoader.getWordSet(Reader, String, Version) directly

      • Fields inherited from class org.apache.lucene.analysis.StopwordAnalyzerBase

        matchVersion, stopwords

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected org.apache.lucene.analysis.ReusableAnalyzerBase.TokenStreamComponents createComponents​(String fieldName, Reader reader)
      Creates ReusableAnalyzerBase.TokenStreamComponents used to tokenize all the text in the provided Reader.
      static Set<?> getDefaultStopSet()
      Returns an unmodifiable instance of the default stop-words set.

      • Methods inherited from class org.apache.lucene.analysis.StopwordAnalyzerBase

        getStopwordSet, loadStopwordSet, loadStopwordSet, loadStopwordSet

      • Methods inherited from class org.apache.lucene.analysis.ReusableAnalyzerBase

        initReader, reusableTokenStream, tokenStream

      • Methods inherited from class org.apache.lucene.analysis.Analyzer

        close, getOffsetGap, getPositionIncrementGap, getPreviousTokenStream, setPreviousTokenStream

    • Field Detail

      • DEFAULT_STOPWORD_FILE

        public static final String DEFAULT_STOPWORD_FILE
        File containing default Arabic stopwords. Default stopword list is from http://members.unine.ch/jacques.savoy/clef/index.html The stopword list is BSD-Licensed.
        See Also:
        Constant Field Values

      • STOPWORDS_COMMENT

        @Deprecated
public static final String STOPWORDS_COMMENT
        Deprecated.
        use WordlistLoader.getWordSet(Reader, String, Version) directly
        The comment character in the stopwords file. All lines prefixed with this will be ignored
        See Also:
        Constant Field Values

    • Constructor Detail

      • ArabicAnalyzer

        public ArabicAnalyzer​(org.apache.lucene.util.Version matchVersion)
        Builds an analyzer with the default stop words: DEFAULT_STOPWORD_FILE.

      • ArabicAnalyzer

        public ArabicAnalyzer​(org.apache.lucene.util.Version matchVersion,
                      Set<?> stopwords)
        Builds an analyzer with the given stop words
        Parameters:
        matchVersion - lucene compatibility version
        stopwords - a stopword set

      • ArabicAnalyzer

        public ArabicAnalyzer​(org.apache.lucene.util.Version matchVersion,
                      Set<?> stopwords,
                      Set<?> stemExclusionSet)
        Builds an analyzer with the given stop word. If a none-empty stem exclusion set is provided this analyzer will add a KeywordMarkerFilter before ArabicStemFilter.
        Parameters:
        matchVersion - lucene compatibility version
        stopwords - a stopword set
        stemExclusionSet - a set of terms not to be stemmed

    • Method Detail

      • getDefaultStopSet

        public static Set<?> getDefaultStopSet()
        Returns an unmodifiable instance of the default stop-words set.
        Returns:
        an unmodifiable instance of the default stop-words set.

      • createComponents

        protected org.apache.lucene.analysis.ReusableAnalyzerBase.TokenStreamComponents createComponents​(String fieldName,
                                                                                                 Reader reader)
        Creates ReusableAnalyzerBase.TokenStreamComponents used to tokenize all the text in the provided Reader.
        Specified by:
        createComponents in class org.apache.lucene.analysis.ReusableAnalyzerBase
        Returns:
        ReusableAnalyzerBase.TokenStreamComponents built from an StandardTokenizer filtered with LowerCaseFilter, StopFilter, ArabicNormalizationFilter, KeywordMarkerFilter if a stem exclusion set is provided and ArabicStemFilter.