Package org.apache.lucene.index.pruning

Class TFTermPruningPolicy

  • public class TFTermPruningPolicy
extends TermPruningPolicy
    Policy for producing smaller index out of an input index, by removing postings data for those terms where their in-document frequency is below a specified threshold.

    Larger threshold value will produce a smaller index. See TermPruningPolicy for size vs performance considerations.

    This implementation uses simple term frequency thresholds to remove all postings from documents where a given term occurs rarely (i.e. its TF in a document is smaller than the threshold).

    Threshold values in this method are expressed as absolute term frequencies.

    • Field Detail

      • defThreshold

        protected int defThreshold

      • curThr

        protected int curThr

    • Constructor Detail

      • TFTermPruningPolicy

        public TFTermPruningPolicy​(org.apache.lucene.index.IndexReader in,
                           Map<String,​Integer> fieldFlags,
                           Map<String,​Integer> thresholds,
                           int defThreshold)

    • Method Detail

      • pruneTermEnum

        public boolean pruneTermEnum​(org.apache.lucene.index.TermEnum te)
                      throws IOException
        Description copied from class: TermPruningPolicy
        Pruning of all postings for a term (invoked once per term).
        Specified by:
        pruneTermEnum in class TermPruningPolicy
        Parameters:
        te - positioned term enum.
        Returns:
        true if all postings for this term should be removed, false otherwise.
        Throws:
        IOException

      • initPositionsTerm

        public void initPositionsTerm​(org.apache.lucene.index.TermPositions in,
                              org.apache.lucene.index.Term t)
                       throws IOException
        Description copied from class: TermPruningPolicy
        Called when moving TermPositions to a new Term.
        Specified by:
        initPositionsTerm in class TermPruningPolicy
        Parameters:
        in - input term positions
        t - current term
        Throws:
        IOException

      • pruneAllPositions

        public boolean pruneAllPositions​(org.apache.lucene.index.TermPositions termPositions,
                                 org.apache.lucene.index.Term t)
                          throws IOException
        Description copied from class: TermPruningPolicy
        Prune all postings per term (invoked once per term per doc)
        Specified by:
        pruneAllPositions in class TermPruningPolicy
        Parameters:
        termPositions - positioned term positions. Implementations MUST NOT advance this by calling TermPositions methods that advance either the position pointer (next, skipTo) or term pointer (seek).
        t - current term
        Returns:
        true if the current posting should be removed, false otherwise.
        Throws:
        IOException

      • pruneTermVectorTerms

        public int pruneTermVectorTerms​(int docNumber,
                                String field,
                                String[] terms,
                                int[] freqs,
                                org.apache.lucene.index.TermFreqVector tfv)
                         throws IOException
        Description copied from class: TermPruningPolicy
        Pruning of individual terms in term vectors.
        Specified by:
        pruneTermVectorTerms in class TermPruningPolicy
        Parameters:
        docNumber - document number
        field - field name
        terms - array of terms
        freqs - array of term frequencies
        tfv - the original term frequency vector
        Returns:
        0 if no terms are to be removed, positive number to indicate how many terms need to be removed. The same number of entries in the terms array must be set to null to indicate which terms to remove.
        Throws:
        IOException

      • pruneSomePositions

        public int pruneSomePositions​(int docNum,
                              int[] positions,
                              org.apache.lucene.index.Term curTerm)
        Description copied from class: TermPruningPolicy
        Prune some postings per term (invoked once per term per doc).
        Specified by:
        pruneSomePositions in class TermPruningPolicy
        Parameters:
        docNum - current document number
        positions - original term positions in the document (and indirectly term frequency)
        curTerm - current term
        Returns:
        0 if no postings are to be removed, or positive number to indicate how many postings need to be removed. The same number of entries in the positions array must be set to -1 to indicate which positions to remove.