org.terrier.structures.indexing

Class LexiconBuilder

java.lang.Object

org.terrier.structures.indexing.LexiconBuilder

public class LexiconBuilder
extends Object

Builds temporary lexicons during indexing a collection and merges them when the indexing of a collection has finished.

Author:

Craig Macdonald & Vassilis Plachouras

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LexiconBuilder.BasicLexiconCollectionStaticticsCounter counts global statistics in the non-fields case
static interface	LexiconBuilder.CollectionStatisticsCounter Counter of LexiconEntries
protected static class	LexiconBuilder.FieldLexiconCollectionStaticticsCounter counts global statistics in the fields case
protected static class	LexiconBuilder.NullCollectionStatisticsCounter

Field Summary

Fields

Modifier and Type	Field and Description
protected String	defaultStructureName
protected int	DocCount How many documents have been processed so far.
protected static int	DocumentsPerLexicon The number of documents for which a temporary lexicon is created.
protected IndexOnDisk	index
protected String	indexPath The directory to write the final lexicons to
protected String	indexPrefix The filename of the lexicons.
protected String	lexiconEntryFactoryValueClass
protected Class<? extends LexiconOutputStream>	lexiconOutputStream class to be used as a lexiconoutpustream.
protected static org.apache.log4j.Logger	logger The logger used for this class
protected static int	MAXLEXMERGE Number of lexicons to merge at once.
protected static boolean	MERGE2LEXATTIME Should we only merge lexicons in pairs (Terrier 1.0.x scheme)? Set by property lexicon.builder.merge.2lex.attime
protected LexiconMap	TempLex The lexicontree to write the current term stream to
protected int	TempLexCount How many temporary lexicons have been generated so far
protected LinkedList<String>	tempLexFiles The linkedlist in which the temporary lexicon structure names are stored.
protected int	TermCount How many terms are in the final lexicon
protected FixedSizeWriteableFactory<LexiconEntry>	valueFactory

Constructor Summary

Constructors

Constructor and Description
LexiconBuilder(IndexOnDisk i, String _structureName) constructor
LexiconBuilder(IndexOnDisk i, String _structureName, Class<? extends LexiconMap> _LexiconMapClass, String _lexiconEntryClass) constructor
LexiconBuilder(IndexOnDisk i, String _structureName, LexiconMap lexiconMap, String _lexiconEntryClass) constructor

Method Summary

Methods

Modifier and Type	Method and Description
void	addDocumentTerms(DocumentPostingList terms) adds the terms of a document to the temporary lexicon in memory.
void	addTemporaryLexicon(String structureName) Deprecated.
void	addTerm(String term, int tf) Add a single term to the lexicon being built
static void	createLexiconHash(IndexOnDisk index) Deprecated. use optimise instead
static void	createLexiconIndex(IndexOnDisk index) Deprecated. use optimise instead
void	finishedDirectIndexBuild() Processing the lexicon after finished creating the direct and document indexes.
void	finishedInvertedIndexBuild() Processing the lexicon after finished creating the inverted index.
void	flush() Force a temporary lexicon to be flushed
int	getFinalNumberOfTerms() Returns the number of terms in the final lexicon.
protected Iterator<Map.Entry<String,LexiconEntry>>	getLexInputStream(String structureName) return the lexicon input stream for the current index at the specified filename
protected LexiconOutputStream<String>	getLexOutputStream(String structureName) return the lexicon outputstream for the current index at the specified filename
protected static LexiconMap	instantiate(Class<? extends LexiconMap> LexiconMapClass)
void	merge(LinkedList<String> filesToMerge) Merges the intermediate lexicon files created during the indexing.
protected void	mergeNLexicons(Iterator<Map.Entry<String,LexiconEntry>>[] lis, LexiconOutputStream<String> los)
protected void	mergeTwoLexicons(Iterator<Map.Entry<String,LexiconEntry>> lis1, Iterator<Map.Entry<String,LexiconEntry>> lis2, LexiconOutputStream<String> los) Merge the two LexiconInputStreams into the given LexiconOutputStream
protected LexiconEntry	newLexiconEntry(int termid)
static void	optimise(IndexOnDisk index, String structureName) Optimises the lexicon, eg lexid file
void	optimiseLexicon() optimise the lexicon
static void	reAssignTermIds(IndexOnDisk index, String structureName, int numEntries) Re-assigned the termids within the named lexicon structure to be ascending with descending term frequency, i.e.
protected void	writeTemporaryLexicon() Writes the current contents of TempLex temporary lexicon binary tree down to a temporary disk lexicon.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

lexiconOutputStream

protected Class<? extends LexiconOutputStream> lexiconOutputStream

class to be used as a lexiconoutpustream. set by this and child classes

lexiconEntryFactoryValueClass

protected final String lexiconEntryFactoryValueClass

logger

protected static final org.apache.log4j.Logger logger

The logger used for this class

DocCount

protected int DocCount

How many documents have been processed so far.

TermCount

protected int TermCount

How many terms are in the final lexicon

DocumentsPerLexicon

protected static final int DocumentsPerLexicon

The number of documents for which a temporary lexicon is created. Corresponds to property bundle.size, default value 2000.

tempLexFiles

protected final LinkedList<String> tempLexFiles

The linkedlist in which the temporary lexicon structure names are stored. These are merged into a single Lexicon by the merge() method. LinkedList is best List implementation for this, as all operations are either append element, or remove first element - making LinkedList ideal.

TempLex

protected LexiconMap TempLex

The lexicontree to write the current term stream to

indexPath

protected String indexPath

The directory to write the final lexicons to

indexPrefix

protected String indexPrefix

The filename of the lexicons.

index

protected IndexOnDisk index

TempLexCount

protected int TempLexCount

How many temporary lexicons have been generated so far

MERGE2LEXATTIME

protected static final boolean MERGE2LEXATTIME

Should we only merge lexicons in pairs (Terrier 1.0.x scheme)? Set by property lexicon.builder.merge.2lex.attime

MAXLEXMERGE

protected static final int MAXLEXMERGE

Number of lexicons to merge at once. Set by property lexicon.builder.merge.lex.max, defaults to 16

defaultStructureName

protected String defaultStructureName

valueFactory

protected FixedSizeWriteableFactory<LexiconEntry> valueFactory

Constructor Detail

LexiconBuilder

public LexiconBuilder(IndexOnDisk i,
              String _structureName)

constructor

Parameters:

i -

_structureName -

LexiconBuilder

public LexiconBuilder(IndexOnDisk i,
              String _structureName,
              Class<? extends LexiconMap> _LexiconMapClass,
              String _lexiconEntryClass)

constructor

Parameters:

i -

_structureName -

_LexiconMapClass -

_lexiconEntryClass -

LexiconBuilder

public LexiconBuilder(IndexOnDisk i,
              String _structureName,
              LexiconMap lexiconMap,
              String _lexiconEntryClass)

constructor

Parameters:

i -

_structureName -

lexiconMap -

_lexiconEntryClass -

Method Detail

instantiate

protected static LexiconMap instantiate(Class<? extends LexiconMap> LexiconMapClass)

getFinalNumberOfTerms

public int getFinalNumberOfTerms()

Returns the number of terms in the final lexicon. Only updated once finishDirectIndexBuild() has executed

addTemporaryLexicon

public void addTemporaryLexicon(String structureName)

Deprecated.

If the application code generated lexicons itself, use this method to add them to the merge list Otherwise dont touch this method.

Parameters:

structureName - Fully path to a lexicon to merge

writeTemporaryLexicon

protected void writeTemporaryLexicon()

Writes the current contents of TempLex temporary lexicon binary tree down to a temporary disk lexicon.

addTerm

public void addTerm(String term,
           int tf)

Add a single term to the lexicon being built

Parameters:

term - The String term

tf - the frequency of the term

addDocumentTerms

public void addDocumentTerms(DocumentPostingList terms)

adds the terms of a document to the temporary lexicon in memory.

Parameters:

terms - DocumentPostingList the terms of the document to add to the temporary lexicon

flush

public void flush()

Force a temporary lexicon to be flushed

finishedInvertedIndexBuild

public void finishedInvertedIndexBuild()

Processing the lexicon after finished creating the inverted index.

finishedDirectIndexBuild

public void finishedDirectIndexBuild()

Processing the lexicon after finished creating the direct and document indexes.

merge

public void merge(LinkedList<String> filesToMerge)
           throws IOException

Merges the intermediate lexicon files created during the indexing.

Parameters:

filesToMerge - java.util.LinkedList the list containing the filenames of the temporary files.

Throws:

IOException - an input/output exception is throws if a problem is encountered.

newLexiconEntry

protected LexiconEntry newLexiconEntry(int termid)

mergeNLexicons

protected void mergeNLexicons(Iterator<Map.Entry<String,LexiconEntry>>[] lis,
                  LexiconOutputStream<String> los)
                       throws IOException

Throws:

IOException

mergeTwoLexicons

protected void mergeTwoLexicons(Iterator<Map.Entry<String,LexiconEntry>> lis1,
                    Iterator<Map.Entry<String,LexiconEntry>> lis2,
                    LexiconOutputStream<String> los)
                         throws IOException

Merge the two LexiconInputStreams into the given LexiconOutputStream

Parameters:

lis1 - First lexicon to be merged

lis2 - Second lexicon to be merged

los - Lexion to be merged to

Throws:

IOException

createLexiconIndex

public static void createLexiconIndex(IndexOnDisk index)
                               throws IOException

Deprecated. use optimise instead

Creates a lexicon index for the specified index

Parameters:

index - IndexOnDisk to make the lexicon index for

Throws:

IOException

createLexiconHash

public static void createLexiconHash(IndexOnDisk index)
                              throws IOException

Deprecated. use optimise instead

Creates a lexicon hash for the specified index

Parameters:

index - IndexOnDisk to make the LexiconHash the lexicoin

Throws:

IOException

optimiseLexicon

public void optimiseLexicon()

optimise the lexicon

optimise

public static void optimise(IndexOnDisk index,
            String structureName)

Optimises the lexicon, eg lexid file

reAssignTermIds

public static void reAssignTermIds(IndexOnDisk index,
                   String structureName,
                   int numEntries)
                            throws IOException

Re-assigned the termids within the named lexicon structure to be ascending with descending term frequency, i.e. the terms with termid 0 will have the highest frequency.

Parameters:

index -

structureName -

numEntries -

Throws:

IOException

getLexInputStream

protected Iterator<Map.Entry<String,LexiconEntry>> getLexInputStream(String structureName)
                                                              throws IOException

return the lexicon input stream for the current index at the specified filename

Throws:

IOException

getLexOutputStream

protected LexiconOutputStream<String> getLexOutputStream(String structureName)
                                                  throws IOException

return the lexicon outputstream for the current index at the specified filename

Throws:

IOException