Configuring Indexing in Terrier |
Firstly, the Collection and Document object parse the collection. For TrecTerrier, you can configure the object used to parse the collection by setting the property trec.collection.class. Additionally, you can set documents that TRECCollection should skip, by adding their docnos to the file named by the property trec.blacklist.docids. Moreover, you can configure Terrier to accept larger terms (by the property max.term.length - default 20), or documents with larger docnos (docno.byte.length - default 20). These two properties replace the string.byte.length property of Terrier 1.0.2.
TRECCollection can be further configured. Set TrecDocTags.doctag to denote the marker tag for document boundaries. TrecDocTags.idtag denotes the tag that contains the DOCNO of the document. TrecDocTags.skip denotes tags that should not be parsed in this collection (for instance, the DOCHDR tags of TREC Web collections). Note that as of Terrier 1.1.0, the specified tags are case-sensitive.
Terrier has the ability to record the occurrences of terms in fields of documents. For TRECCollection, you can specify the fields that should be recorded by the FieldTags.process property. For example, to note when a term occurs in the TITLE or H1 HTML tags of a documents, set FieldTags.process=TITLE,H1.
The indexer iterates through the documents of the collection and sends each term found through the TermPipeline. The TermPipeline transforms the terms, and can remove terms that should not be indexed. The TermPipeline chain in use is termpipelines=Stopwords,PorterStemmer, which removes terms from the document using the Stopwords object, and then applies Porter's Stemming algorithm for English to the terms (PorterStemmer). If you wanted to use a different stemmer, this is the point at which it should be called.
The indexers are more complicated. Each class can be configured by several properties. Many of these alter the memory usage of the classes.
Once terms have been processed through the TermPipeline, they are aggregated by the DocumentPostingList and the LexiconMap. These have a few properties:
The LexiconMap is flushed to disk every bundle.size documents. If memory during indexing is a concern, then reduce this property to less than its default 2500. However, more temporary lexicons will be created. The rate at which the temporary lexicons are merged is controlled by the lexicon.builder.merge.lex.max property, though we have found 16 to be a good compromise. Finally, if you set lexicon.use.hash to true, then Lexicon read performance will be enhanced by the creation of a lexicon hash, which reduces the size of the binary search when reading the lexicon for a term (i.e. findTerm(String)).
Once all documents in the index have been created, the InvertedIndex is created by the InvertedIndexBuilder. As the entire DirectIndex cannot be inverted in memory, the InvertedIndexBuilder takes several iterations, selecting a few terms, scanning the direct index for them, and then writing out their postings to the inverted index. If it takes too many terms at once, Terrier can run out of memory. Reduce the property invertedfile.processpointers from its default 20,000,000 and rerun (default is only 2,000,000 for block indexing, which is more memory intensive). See the InvertedIndexBuilder for more information about the inversion and term selection strategies.
A block is a unit of text in a document. When you index using blocks, you tell Terrier to save positional information with each term. Depending on how Terrier has been configured, a block can be of size 1 or larger. Size 1 means that the exact position of each term can be determined. For size > 1, the block id is incremented after every N terms. You can configure the size of a block using the property block.size.
You can enable block indexing by setting the property block.indexing to true in your terrier.properties file. This ensures that the Indexer used for indexing is the BlockIndexer, not the BasicIndexer. When loading an index, Terrier will detect that the index has block information saved and use the appropriate classes for reading the index files.
You can use the positional information when doing retrieval. For instance, you search for documents matching a phrase, e.g. "Terabyte retriever", or where the words occur near each other, e.g. "indexing blocks"~20.
When you enable the property block.indexing, the indexer used is the BlockIndexer, not the BasicIndexer. The DirectIndex and InvertedIndex created use a different format, which includes the blockids for each posting, and can be read by BlockDirectIndex and BlockInvertedIndex respectively. During indexing, BlockLexicons are created that keep track of how many blocks are in use for a term. However, at the last stage of rewriting the lexicon at the end of inverted indexing, the BlockLexicon is rewritten as a normal Lexicon, as the block information can be guessed during retrieval.
Copyright © 2011 University of Glasgow | All Rights Reserved
