org.terrier.matching.dsms

Class DependenceScoreModifier

java.lang.Object

org.terrier.matching.dsms.DependenceScoreModifier

All Implemented Interfaces:

Cloneable, DocumentScoreModifier

Direct Known Subclasses:

DFRDependenceScoreModifier, MRFDependenceScoreModifier

public abstract class DependenceScoreModifier
extends Object
implements DocumentScoreModifier

Base class for Dependence models. Document scores are modified using n-grams, approximating the dependence of terms between documents. Implemented as a document score modifier, similarly to PhraseScoreModifier. Postings lists are traversed in a DAAT fashion.

Properties

• proximity.dependency.type - one of SD, FD for sequential dependence or full dependence
• proximity.ngram.length - proxmity windows, in tokens
• proximity.w_t - weight of unigram in combination, defaults 1.0d
• proximity.w_o - weight of SD in combination, default 1.0d
• proximity.w_u - weight of FD in combination, default 1.0d
• proximity.qtw.fnid - combination function to combine the qtws of two terms involved in a phrase. See below.

QTW Combination Functions

1. 1: phraseQTW = 0.5 * (qtw1 + qtw2)
2. 2: phraseQTW = qtw1 * qtw2
3. 3: phraseQTW = min(qtw1, qtw2)
4. 4: phraseQTW = max(qtw1, qtw2)

Since:

3.0

Author:

Craig Macdonald, Vassilis Plachouras, Jie Peng

Field Summary

Fields

Modifier and Type	Field and Description
protected double	avgDocLen
protected String	dependency type of proximity to use
protected int	ngramLength The size of the considered ngrams
protected double	numTokens
protected int	phraseQTWfnid
protected String[]	phraseTerms A list of the strings of the phrase terms.
protected double	w_o weight of ordered dependence model
protected double	w_t weight of unigram model
protected double	w_u weight of unordered dependence model

Constructor Summary

Constructors

Constructor and Description
DependenceScoreModifier() Constructs an instance of the DependenceScoreModifier.

Method Summary

Modifier and Type	Method and Description
protected double	calculateDependence(Posting[] ips, boolean[] okToUse, double[] phraseTermWeights, boolean SD) calculates the dependence score for one document, using the IterablePostings available.
Object	clone() Creates a clone of this object
protected static int	countTrue(boolean[] in)
protected void	determineGlobalStatistics(String[] terms, EntryStatistics[] es, boolean SD) unused hook method
protected void	doDependency(Index index, EntryStatistics[] es, IterablePosting[] ips, ResultSet rs, double[] phraseTermWeights, boolean SD) Calculates dependence scores for all documents, putting the scores into the ResultSet rs
String	getName() Returns the name of the modifier.
boolean	modifyScores(Index index, MatchingQueryTerms terms, ResultSet set) Modifies the scores of documents, in which there exist, or there does not exist a given phrase.
protected static boolean	NOR(boolean[] in)
protected void	openPostingLists(Index index, LexiconEntry[] les, IterablePosting[] ips) Opens the posting list for an index and lexicon entry
double	score(Posting[] postings) Calculate the score for a document (from the given posting for that document)
protected double	scoreFDSD(boolean SD, int i, Posting ip1, int j, Posting ip2, double _avgDocLen) how likely is it that these two postings have so many near-occurrences, given the length of this document
protected abstract double	scoreFDSD(int matchingNGrams, int docLength)
void	setCollectionStatistics(CollectionStatistics cs, Index _index) Sets the collection statistics used to score the documents (number of documents in the collection, etc)

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ngramLength

protected int ngramLength

The size of the considered ngrams

dependency

protected String dependency

type of proximity to use

phraseQTWfnid

protected final int phraseQTWfnid

w_t

protected double w_t

weight of unigram model

w_o

protected double w_o

weight of ordered dependence model

w_u

protected double w_u

weight of unordered dependence model

phraseTerms

protected String[] phraseTerms

A list of the strings of the phrase terms.

avgDocLen

protected double avgDocLen

numTokens

protected double numTokens

Constructor Detail

DependenceScoreModifier

public DependenceScoreModifier()

Constructs an instance of the DependenceScoreModifier.

Method Detail

clone

public Object clone()

Creates a clone of this object

Specified by:

clone in interface DocumentScoreModifier

Overrides:

clone in class Object

scoreFDSD

protected abstract double scoreFDSD(int matchingNGrams,
                                    int docLength)

getName

public String getName()

Returns the name of the modifier.

Specified by:

getName in interface DocumentScoreModifier

Returns:

String the name of the modifier.

NOR

protected static boolean NOR(boolean[] in)

modifyScores

public boolean modifyScores(Index index,
                            MatchingQueryTerms terms,
                            ResultSet set)

Modifies the scores of documents, in which there exist, or there does not exist a given phrase.

Specified by:

modifyScores in interface DocumentScoreModifier

Parameters:

index - Index the data structures to use.

terms - MatchingQueryTerms the terms to be matched for the query. This does not correspond to the phrase terms necessarily, but to all the terms of the query.

set - ResultSet the result set for the query.

Returns:

true if any scores have been altered

determineGlobalStatistics

protected void determineGlobalStatistics(String[] terms,
                                         EntryStatistics[] es,
                                         boolean SD)
                                  throws IOException

unused hook method

Throws:

IOException

openPostingLists

protected void openPostingLists(Index index,
                                LexiconEntry[] les,
                                IterablePosting[] ips)
                         throws IOException

Opens the posting list for an index and lexicon entry

Throws:

IOException

doDependency

protected void doDependency(Index index,
                            EntryStatistics[] es,
                            IterablePosting[] ips,
                            ResultSet rs,
                            double[] phraseTermWeights,
                            boolean SD)
                     throws IOException

Calculates dependence scores for all documents, putting the scores into the ResultSet rs

Throws:

IOException

calculateDependence

protected double calculateDependence(Posting[] ips,
                                     boolean[] okToUse,
                                     double[] phraseTermWeights,
                                     boolean SD)

calculates the dependence score for one document, using the IterablePostings available.

Parameters:

ips - all of the IterablePostings

okToUse - the IterablePostings that are set on the current document

phraseTermWeights - weights on each of the query terms

SD - is sequential dependence to be used

Returns:

score of this dependence score modifier for the current document

countTrue

protected static int countTrue(boolean[] in)

setCollectionStatistics

public void setCollectionStatistics(CollectionStatistics cs,
                                    Index _index)

Sets the collection statistics used to score the documents (number of documents in the collection, etc)

score

public double score(Posting[] postings)

Calculate the score for a document (from the given posting for that document)

scoreFDSD

protected double scoreFDSD(boolean SD,
                           int i,
                           Posting ip1,
                           int j,
                           Posting ip2,
                           double _avgDocLen)

how likely is it that these two postings have so many near-occurrences, given the length of this document