CompressingMetaIndexBuilder (Terrier Information Retrieval Platform 5.1 API)

java.lang.Object
- org.terrier.structures.indexing.MetaIndexBuilder
- - org.terrier.structures.indexing.CompressingMetaIndexBuilder

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable
```
public class CompressingMetaIndexBuilder
extends MetaIndexBuilder
implements Flushable
```
Creates a metaindex structure that compresses all values using Deflator. Properties:
- metaindex.compressed.max.data.in-mem.mb - maximum size that a meta index .zdata file will be kept in memory. Defaults to 400(mb).
- metaindex.compressed.max.index.in-mem.mb - maximum size that a meta index .zdata file will be kept in memory. Defaults to 100(mb).
- metaindex.compressed.reverse.allow.duplicates - set this property to true to suppress errors when a reverse meta value is not unique. Default false.
- metaindex.compressed.crop.long - set this property to suppress errors with overlong Document metadata, while will instead be cropped.
Since:

3.0

Author:

Craig Macdonald & Vassilis Plachouras

Field Summary

Fields
Modifier and Type	Field and Description
`protected ByteArrayOutputStream`	`baos`
`protected byte[]`	`compressedBuffer`
`protected boolean`	`CROP_LONG`
`protected long`	`currentIndexOffset`
`protected long`	`currentOffset`
`protected DataOutputStream`	`dataOutput`
`protected int`	`DOCS_PER_CHECK`
`protected int`	`entryCount`
`protected int`	`entryLengthBytes`
`protected String[]`	`forwardKeyNames`
`protected int[]`	`forwardKeys`
`protected boolean[]`	`forwardKeyValuesSorted`
`protected FSOrderedMapFile.MapFileWriter[]`	`forwardWriters`
`protected IndexOnDisk`	`index`
`protected DataOutputStream`	`indexOutput`
`protected gnu.trove.TObjectIntHashMap<String>`	`key2Index`
`protected int`	`keyCount`
`protected FixedSizeWriteableFactory<org.apache.hadoop.io.Text>[]`	`keyFactories`
`protected String[]`	`keyNames`
`protected String[]`	`lastValues`
`protected org.slf4j.Logger`	`logger`
`protected int`	`MAX_INDEX_MB_IN_MEM_RETRIEVAL`
`protected int`	`MAX_MB_IN_MEM_RETRIEVAL`
`protected MemoryChecker`	`memCheck`
`protected boolean`	`REVERSE_ALLOW_DUPS`
`protected int`	`REVERSE_KEY_LOOKUP_WRITING_BUFFER_SIZE`
`protected byte[]`	`spaces`
`protected String`	`structureName`
`protected int[]`	`valueLensBytes`
`protected int[]`	`valueLensChars`
`protected Deflater`	`zip`
`protected int`	`ZIP_COMPRESSION_LEVEL`

Constructor Summary

Constructors
Constructor and Description
`CompressingMetaIndexBuilder(IndexOnDisk _index, String[] _keyNames, int[] _valueLens, String[] _forwardKeys)` constructor
`CompressingMetaIndexBuilder(IndexOnDisk _index, String _structureName, String[] _keyNames, int[] _valueLens, String[] _forwardKeys)` constructor

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`close()`
`void`	`flush()`
`void`	`writeDocumentEntry(Map<String,String> data)` Write out metadata for current document, extracted from specified map Typically, the MetaIndexBuilder will know which keys from data that it is interested in.
`void`	`writeDocumentEntry(String[] data)` Write out metadata for current document.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected final org.slf4j.Logger logger

MAX_MB_IN_MEM_RETRIEVAL

protected final int MAX_MB_IN_MEM_RETRIEVAL

MAX_INDEX_MB_IN_MEM_RETRIEVAL

protected final int MAX_INDEX_MB_IN_MEM_RETRIEVAL

REVERSE_ALLOW_DUPS

protected final boolean REVERSE_ALLOW_DUPS

CROP_LONG
```
protected final boolean CROP_LONG
```

REVERSE_KEY_LOOKUP_WRITING_BUFFER_SIZE
```
protected final int REVERSE_KEY_LOOKUP_WRITING_BUFFER_SIZE
```
See Also:

Constant Field Values

DOCS_PER_CHECK
```
protected final int DOCS_PER_CHECK
```

ZIP_COMPRESSION_LEVEL
```
protected final int ZIP_COMPRESSION_LEVEL
```
See Also:

Constant Field Values

key2Index

protected final gnu.trove.TObjectIntHashMap<String> key2Index

dataOutput
```
protected DataOutputStream dataOutput
```

keyNames
```
protected final String[] keyNames
```

keyCount
```
protected final int keyCount
```

zip
```
protected Deflater zip
```

baos
```
protected ByteArrayOutputStream baos
```

indexOutput
```
protected DataOutputStream indexOutput
```

compressedBuffer
```
protected byte[] compressedBuffer
```

index
```
protected IndexOnDisk index
```

valueLensChars
```
protected int[] valueLensChars
```

valueLensBytes
```
protected int[] valueLensBytes
```

spaces
```
protected byte[] spaces
```

entryLengthBytes
```
protected int entryLengthBytes
```

currentOffset
```
protected long currentOffset
```

currentIndexOffset
```
protected long currentIndexOffset
```

entryCount
```
protected int entryCount
```

forwardKeys
```
protected int[] forwardKeys
```

forwardKeyNames
```
protected String[] forwardKeyNames
```

forwardWriters

protected FSOrderedMapFile.MapFileWriter[] forwardWriters

forwardKeyValuesSorted

protected boolean[] forwardKeyValuesSorted

lastValues
```
protected String[] lastValues
```

memCheck
```
protected MemoryChecker memCheck
```

keyFactories

protected FixedSizeWriteableFactory<org.apache.hadoop.io.Text>[] keyFactories

structureName
```
protected String structureName
```

Constructor Detail

CompressingMetaIndexBuilder

public CompressingMetaIndexBuilder(IndexOnDisk _index,
                                   String[] _keyNames,
                                   int[] _valueLens,
                                   String[] _forwardKeys)

constructor

Parameters:: _index -; _keyNames -; _valueLens -; _forwardKeys -

CompressingMetaIndexBuilder

public CompressingMetaIndexBuilder(IndexOnDisk _index,
                                   String _structureName,
                                   String[] _keyNames,
                                   int[] _valueLens,
                                   String[] _forwardKeys)

constructor

Parameters:: _index -; _structureName -; _keyNames -; _valueLens -; _forwardKeys -

Method Detail
- writeDocumentEntry
```
public void writeDocumentEntry(Map<String,String> data)
                        throws IOException
```
  Write out metadata for current document, extracted from specified map Typically, the MetaIndexBuilder will know which keys from data that it is interested in.
  
  Specified by:
  
  writeDocumentEntry in class MetaIndexBuilder
  
  Throws:
  
  IOException
- writeDocumentEntry
```
public void writeDocumentEntry(String[] data)
                        throws IOException
```
  Write out metadata for current document. Values for all keys are specified.
  
  Specified by:
  
  writeDocumentEntry in class MetaIndexBuilder
  
  Throws:
  
  IOException
- flush
```
public void flush()
           throws IOException
```
  Specified by:
  
  flush in interface Flushable
  
  Throws:
  
  IOException
- close
```
public void close()
           throws IOException
```
  Specified by:
  
  close in interface Closeable
  
  Specified by:
  
  close in interface AutoCloseable
  
  Throws:
  
  IOException

Class CompressingMetaIndexBuilder

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

logger

MAX_MB_IN_MEM_RETRIEVAL

MAX_INDEX_MB_IN_MEM_RETRIEVAL

REVERSE_ALLOW_DUPS

CROP_LONG

REVERSE_KEY_LOOKUP_WRITING_BUFFER_SIZE

DOCS_PER_CHECK

ZIP_COMPRESSION_LEVEL

key2Index

dataOutput

keyNames

keyCount

zip

baos

indexOutput

compressedBuffer

index

valueLensChars

valueLensBytes

spaces

entryLengthBytes

currentOffset

currentIndexOffset

entryCount

forwardKeys

forwardKeyNames

forwardWriters

forwardKeyValuesSorted

lastValues

memCheck

keyFactories

structureName

Constructor Detail

CompressingMetaIndexBuilder

CompressingMetaIndexBuilder

Method Detail

writeDocumentEntry

writeDocumentEntry

flush

close