org.terrier.indexing
Class BlockSinglePassIndexer

java.lang.Object
  extended by org.terrier.indexing.Indexer
      extended by org.terrier.indexing.BasicIndexer
          extended by org.terrier.indexing.BasicSinglePassIndexer
              extended by org.terrier.indexing.BlockSinglePassIndexer

public class BlockSinglePassIndexer
extends BasicSinglePassIndexer

Indexes a document collection saving block information for the indexed terms. It performs a single pass inversion (see BasicSinglePassIndexer). All normal block properties are supported. For more information, see BlockIndexer.

Author:
Roi Blanco, Craig Macdonald, Rodrygo Santos.

Nested Class Summary
protected  class BlockSinglePassIndexer.BasicTermProcessor
          This class implements an end of a TermPipeline that adds the term to the DocumentTree.
protected  class BlockSinglePassIndexer.DelimFieldTermProcessor
          This class behaves in a similar fashion to FieldTermProcessor except that this one treats blocks bounded by delimiters instead of fixed-sized blocks.
protected  class BlockSinglePassIndexer.DelimTermProcessor
          This class behaves in a similar fashion to BasicTermProcessor except that this one treats blocks bounded by delimiters instead of fixed-sized blocks.
protected  class BlockSinglePassIndexer.FieldTermProcessor
          This class implements an end of a TermPipeline that adds the term to the DocumentTree.
 
Field Summary
protected  int BLOCK_SIZE
          The maximum number of terms allowed in a block
protected  int blockId
          The block number in the current document.
protected  int MAX_BLOCKS
          The maximum number allowed number of blocks in a document.
protected  int numOfTokensInBlock
          The number of tokens in the current block of the current document.
 
Fields inherited from class org.terrier.indexing.BasicSinglePassIndexer
basicInvertedIndexPostingIteratorClass, currentFile, currentId, docsPerCheck, fieldInvertedIndexPostingIteratorClass, fileNames, invertedIndexClass, invertedIndexInputStreamClass, maxDocsPerFlush, maxMemory, memoryAfterFlush, memoryCheck, merger, mp, numberOfDocsSinceCheck, numberOfDocsSinceFlush, numberOfDocuments, numberOfPointers, numberOfTokens, numberOfUniqueTerms, runtime
 
Fields inherited from class org.terrier.indexing.BasicIndexer
numOfTokensInDocument, termFields, termsInDocument
 
Fields inherited from class org.terrier.indexing.Indexer
basicDirectIndexPostingIteratorClass, BUILDER_BOUNDARY_DOCUMENTS, currentIndex, directIndexBuilder, docIndexBuilder, emptyDocIndexEntry, fieldDirectIndexPostingIteratorClass, fieldNames, fileNameNoExtension, IndexEmptyDocuments, invertedIndexBuilder, lexiconBuilder, logger, MAX_DOCS_PER_BUILDER, MAX_TOKENS_IN_DOCUMENT, metaBuilder, numFields, path, pipeline_first, prefix, useFieldInformation
 
Constructor Summary
BlockSinglePassIndexer(java.lang.String pathname, java.lang.String prefix)
          Constructs an instance of this block indexer which uses the single-pass strategy
 
Method Summary
protected  void createDocumentPostings()
          Hook method that creates the right type of DocumentTree class.
protected  void createFieldRunMerger(java.lang.String[][] files)
          Hook method that creates a FieldRunMerger instance
protected  void createMemoryPostings()
          Hook method that creates the right type of MemoryPostings class.
protected  void createRunMerger(java.lang.String[][] files)
          Hook method that creates a RunsMerger instance
protected  TermPipeline getEndOfPipeline()
          Returns the object that is to be the end of the TermPipeline.
 
Methods inherited from class org.terrier.indexing.BasicSinglePassIndexer
checkFlush, createDirectIndex, createInvertedIndex, createInvertedIndex, finishMemoryPosting, forceFlush, getFileNames, indexDocument, load_indexer_properties, performMultiWayMerge
 
Methods inherited from class org.terrier.indexing.BasicIndexer
finishedInvertedIndexBuild
 
Methods inherited from class org.terrier.indexing.Indexer
createMetaIndexBuilder, finishedDirectIndexBuild, index, indexEmpty, init, load_builder_boundary_documents, load_field_ids, load_pipeline, main, merge, merge, mergeTwoIndices, parseInts, useFieldInformation
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

numOfTokensInBlock

protected int numOfTokensInBlock
The number of tokens in the current block of the current document.


blockId

protected int blockId
The block number in the current document.


BLOCK_SIZE

protected int BLOCK_SIZE
The maximum number of terms allowed in a block


MAX_BLOCKS

protected int MAX_BLOCKS
The maximum number allowed number of blocks in a document. After this value, all the remaining terms are in the final block

Constructor Detail

BlockSinglePassIndexer

public BlockSinglePassIndexer(java.lang.String pathname,
                              java.lang.String prefix)
Constructs an instance of this block indexer which uses the single-pass strategy

Parameters:
pathname - String location of the index
prefix - String prefix to file of the index
Method Detail

getEndOfPipeline

protected TermPipeline getEndOfPipeline()
Returns the object that is to be the end of the TermPipeline. This method is used at construction time of the parent object.

Overrides:
getEndOfPipeline in class BasicIndexer
Returns:
TermPipeline the last component of the term pipeline.

createFieldRunMerger

protected void createFieldRunMerger(java.lang.String[][] files)
                             throws java.io.IOException
Description copied from class: BasicSinglePassIndexer
Hook method that creates a FieldRunMerger instance

Overrides:
createFieldRunMerger in class BasicSinglePassIndexer
Throws:
java.io.IOException - if an I/O error occurs.

createRunMerger

protected void createRunMerger(java.lang.String[][] files)
                        throws java.lang.Exception
Description copied from class: BasicSinglePassIndexer
Hook method that creates a RunsMerger instance

Overrides:
createRunMerger in class BasicSinglePassIndexer
Throws:
java.io.IOException - if an I/O error occurs.
java.lang.Exception

createMemoryPostings

protected void createMemoryPostings()
Description copied from class: BasicSinglePassIndexer
Hook method that creates the right type of MemoryPostings class.

Overrides:
createMemoryPostings in class BasicSinglePassIndexer

createDocumentPostings

protected void createDocumentPostings()
Description copied from class: BasicIndexer
Hook method that creates the right type of DocumentTree class.

Overrides:
createDocumentPostings in class BasicIndexer


Terrier 3.5. Copyright © 2004-2011 University of Glasgow