uk.ac.gla.terrier.indexing
Class Indexer

java.lang.Object
  uk.ac.gla.terrier.indexing.Indexer

Direct Known Subclasses:

BasicIndexer, BlockIndexer

public abstract class Indexer
extends java.lang.Object

Properties:

• termpipelines - the sequence of TermPipeline stages (e.g. Stopwords removal and PorterStemmer).
• termpipelines.skip - a list of tokens which should not be skipped from the term pipeline. If not set or empty, then none will be skipped.
• indexing.max.tokens - The maximum number of tokens the indexer will attempt to index in a document. If 0, then all tokens will be indexed (default).
• ignore.empty.documents - Assign empty documents documnent Ids. Default true
• indexing.max.docs.per.builder - Maximum number of documents in an index before a new index is created, and merged later.
• indexing.builder.boundary.docnos - Docnos of documents that force the index being created to be completed, and a new index to be commenced. An alternative to indexing.max.docs.per.builder

Version:

$Revision: 1.44 $

Author:

Craig Macdonald

Constructor Summary

Indexer()

Indexer(java.lang.String path, java.lang.String prefix)
Creates an instance of the class.

Method Summary

abstract void	createDirectIndex(Collection[] collections) An abstract method for creating the direct index, the document index and the lexicon for the given collections.
abstract void	createInvertedIndex() An abstract method for creating the inverted index, given that the the direct index, the document index and the lexicon have already been created.
void	index(Collection[] collections) Creates the data structures for a set of collections.
boolean	isUTFIndexing()
static void	main(java.lang.String[] args)
static void	merge(java.lang.String mpath, java.lang.String mprefix, int lowest, int highest) Merge a series of numbered indices in the same path/prefix area.
static void	merge(java.lang.String mpath, java.lang.String mprefix, java.util.LinkedList<java.lang.String[]> llist, int counterMerged) Merge a series of indices, in pair-wise fashion
boolean	useFieldInformation()

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Indexer

public Indexer()

Indexer

public Indexer(java.lang.String path,
               java.lang.String prefix)

Creates an instance of the class. The generated data structures will be saved in the given path. The of the data is given by the prefix parameter.

Parameters:

path - String the path where the generated data structures will be saved.

prefix - String the filename that the data structures will have.

Method Detail

createDirectIndex

public abstract void createDirectIndex(Collection[] collections)

An abstract method for creating the direct index, the document index and the lexicon for the given collections.

Parameters:

collections - Collection[] An array of collections to index

createInvertedIndex

public abstract void createInvertedIndex()

An abstract method for creating the inverted index, given that the the direct index, the document index and the lexicon have already been created.

index

public void index(Collection[] collections)

Creates the data structures for a set of collections. It creates a set of data structures for every indexing.max.docs.per.builder, if the value of this property is greater than zero, and then it mertges the generated data structures.

Parameters:

collections - The document collection objects to index.

merge

public static void merge(java.lang.String mpath,
                         java.lang.String mprefix,
                         int lowest,
                         int highest)

Merge a series of numbered indices in the same path/prefix area. New merged index will be stored at mpath/mprefix_highest+1.

Parameters:

mpath - Path of all indices

mprefix - Common prefix of all indices

lowest - lowest subfix of prefix

highest - highest subfix of prefix

merge

public static void merge(java.lang.String mpath,
                         java.lang.String mprefix,
                         java.util.LinkedList<java.lang.String[]> llist,
                         int counterMerged)

Merge a series of indices, in pair-wise fashion

Parameters:

mpath - Common path of all indices

mprefix - Prefix of target index

counterMerged - - number of indices to merge

isUTFIndexing

public boolean isUTFIndexing()

useFieldInformation

public boolean useFieldInformation()

main

public static void main(java.lang.String[] args)