StructureMerger (Terrier Information Retrieval Platform 5.1 API)

java.lang.Object
- org.terrier.structures.merging.StructureMerger

Direct Known Subclasses:

BlockStructureMerger
```
public class StructureMerger
extends Object
```
This class merges the structures created by Terrier, so that we use fewer and larger inverted and direct files.
Properties:
- lexicon.use.hash - build a lexicon hash file for new index. Set to true by default.
- merge.direct - merge the direct indices if both indices have them. Set to true by default.
Author:

Vassilis Plachouras and Craig Macdonald

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class StructureMerger.Command

Nested Classes
Modifier and Type	Class and Description
`static class`	`StructureMerger.Command`

Field Summary

Fields
Modifier and Type	Field and Description
`protected String`	`basicDirectIndexPostingIteratorClass`
`protected String`	`basicInvertedIndexPostingIteratorClass`
`protected CompressionFactory.CompressionConfiguration`	`compressionDirectConfig`
`protected CompressionFactory.CompressionConfiguration`	`compressionInvertedConfig`
`protected IndexOnDisk`	`destIndex` destination index
`protected Class<? extends DirectInvertedOutputStream>`	`directFileOutputStreamClass` class to use to write direct file
`protected int`	`fieldCount`
`protected Class<? extends DirectInvertedOutputStream>`	`fieldDirectFileOutputStreamClass`
`protected String`	`fieldDirectIndexPostingIteratorClass`
`protected String`	`fieldInvertedIndexPostingIteratorClass`
`protected boolean`	`keepTermCodeMap`
`protected static org.slf4j.Logger`	`logger` the logger used
`protected boolean`	`MetaReverse`
`protected int`	`numberOfDocuments` The number of documents in the merged structures.
`protected long`	`numberOfPointers` The number of pointers in the merged structures.
`protected int`	`numberOfTerms` The number of terms in the collection.
`protected IndexOnDisk`	`srcIndex1` source index 1
`protected IndexOnDisk`	`srcIndex2` source index 2
`protected gnu.trove.TIntIntHashMap`	`termcodeHashmap` A hashmap for converting the codes of terms appearing only in the vocabulary of the second set of data structures into a new set of term codes for the merged set of data structures.

Constructor Summary

Constructors
Constructor and Description

StructureMerger(IndexOnDisk _srcIndex1, IndexOnDisk _srcIndex2, IndexOnDisk _destIndex)
constructor

Constructors
Constructor and Description
`StructureMerger(IndexOnDisk _srcIndex1, IndexOnDisk _srcIndex2, IndexOnDisk _destIndex)` constructor

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected void`	`createLexidFile()` creates the final term code to offset file, and the lexicon hash if enabled.
`protected static Class<?>[]`	`getInterfaces(Object o)`
`static void`	`main(String[] args)`
`protected void`	`mergeDirectFiles()` Merges the two direct files and the corresponding document id files.
`protected void`	`mergeDocumentIndexFiles()` Merges the two document index files, and the meta files.
`protected void`	`mergeInvertedFiles()` Merges the two lexicons into one.
`void`	`mergeStructures()` Merges the structures created by terrier.
`void`	`setOutputIndex(IndexOnDisk _outputIndex)` Sets the output index.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - logger
```
protected static final org.slf4j.Logger logger
```
    the logger used
  - termcodeHashmap
```
protected gnu.trove.TIntIntHashMap termcodeHashmap
```
    A hashmap for converting the codes of terms appearing only in the vocabulary of the second set of data structures into a new set of term codes for the merged set of data structures.
  - keepTermCodeMap
```
protected boolean keepTermCodeMap
```
  - numberOfDocuments
```
protected int numberOfDocuments
```
    The number of documents in the merged structures.
  - numberOfPointers
```
protected long numberOfPointers
```
    The number of pointers in the merged structures.
  - numberOfTerms
```
protected int numberOfTerms
```
    The number of terms in the collection.
  - compressionDirectConfig
```
protected CompressionFactory.CompressionConfiguration compressionDirectConfig
```
  - compressionInvertedConfig
```
protected CompressionFactory.CompressionConfiguration compressionInvertedConfig
```
  - MetaReverse
```
protected boolean MetaReverse
```
  - srcIndex1
```
protected IndexOnDisk srcIndex1
```
    source index 1
  - srcIndex2
```
protected IndexOnDisk srcIndex2
```
    source index 2
  - destIndex
```
protected IndexOnDisk destIndex
```
    destination index
  - directFileOutputStreamClass
```
protected Class<? extends DirectInvertedOutputStream> directFileOutputStreamClass
```
    class to use to write direct file
  - fieldDirectFileOutputStreamClass
```
protected Class<? extends DirectInvertedOutputStream> fieldDirectFileOutputStreamClass
```
  - fieldCount
```
protected final int fieldCount
```
  - basicInvertedIndexPostingIteratorClass
```
protected String basicInvertedIndexPostingIteratorClass
```
  - fieldInvertedIndexPostingIteratorClass
```
protected String fieldInvertedIndexPostingIteratorClass
```
  - basicDirectIndexPostingIteratorClass
```
protected String basicDirectIndexPostingIteratorClass
```
  - fieldDirectIndexPostingIteratorClass
```
protected String fieldDirectIndexPostingIteratorClass
```
- Constructor Detail
  - StructureMerger
```
public StructureMerger(IndexOnDisk _srcIndex1,
                       IndexOnDisk _srcIndex2,
                       IndexOnDisk _destIndex)
```
    constructor
    
    Parameters:
    
    _srcIndex1 -
    
    _srcIndex2 -
    
    _destIndex -
- Method Detail
  - setOutputIndex
```
public void setOutputIndex(IndexOnDisk _outputIndex)
```
    Sets the output index. This index should have no documents
    
    Parameters:
    
    _outputIndex - the index to be merged to
  - mergeInvertedFiles
```
protected void mergeInvertedFiles()
```
    Merges the two lexicons into one. After this stage, the offsets in the lexicon are ot correct. They will be updated only after creating the inverted file.
  - mergeDirectFiles
```
protected void mergeDirectFiles()
```
    Merges the two direct files and the corresponding document id files.
  - getInterfaces
```
protected static Class<?>[] getInterfaces(Object o)
```
  - mergeDocumentIndexFiles
```
protected void mergeDocumentIndexFiles()
```
    Merges the two document index files, and the meta files.
  - createLexidFile
```
protected void createLexidFile()
```
    creates the final term code to offset file, and the lexicon hash if enabled.
  - mergeStructures
```
public void mergeStructures()
```
    Merges the structures created by terrier.
  - main
```
public static void main(String[] args)
                 throws Exception
```
    Throws:
    
    Exception

Class StructureMerger

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

logger

termcodeHashmap

keepTermCodeMap

numberOfDocuments

numberOfPointers

numberOfTerms

compressionDirectConfig

compressionInvertedConfig

MetaReverse

srcIndex1

srcIndex2

destIndex

directFileOutputStreamClass

fieldDirectFileOutputStreamClass

fieldCount

basicInvertedIndexPostingIteratorClass

fieldInvertedIndexPostingIteratorClass

basicDirectIndexPostingIteratorClass

fieldDirectIndexPostingIteratorClass

Constructor Detail

StructureMerger

Method Detail

setOutputIndex

mergeInvertedFiles

mergeDirectFiles

getInterfaces

mergeDocumentIndexFiles

createLexidFile

mergeStructures

main