org.terrier.structures.indexing

Class CompressingMetaIndexBuilder

java.lang.Object

org.terrier.structures.indexing.MetaIndexBuilder

org.terrier.structures.indexing.CompressingMetaIndexBuilder

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable

public class CompressingMetaIndexBuilder
extends MetaIndexBuilder
implements Flushable

Creates a metaindex structure that compresses all values using Deflator. Properties:

• metaindex.compressed.max.data.in-mem.mb - maximum size that a meta index .zdata file will be kept in memory. Defaults to 400(mb).
• metaindex.compressed.max.index.in-mem.mb - maximum size that a meta index .zdata file will be kept in memory. Defaults to 100(mb).
• metaindex.compressed.reverse.allow.duplicates - set this property to true to suppress errors when a reverse meta value is not unique. Default false.
• metaindex.compressed.crop.long - set this property to suppress errors with overlong Document metadata, while will instead be cropped.

Since:

3.0

Author:

Craig Macdonald & Vassilis Plachouras

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CompressingMetaIndexBuilder.KeyedPartitioner class KeyedPartitioner

Field Summary

Fields

Modifier and Type	Field and Description
protected ByteArrayOutputStream	baos
protected byte[]	compressedBuffer
protected static boolean	CROP_LONG
protected long	currentIndexOffset
protected long	currentOffset
protected DataOutputStream	dataOutput
protected static int	DOCS_PER_CHECK
protected int	entryCount
protected int	entryLengthBytes
protected String[]	forwardKeyNames
protected int[]	forwardKeys
protected boolean[]	forwardKeyValuesSorted
protected FSOrderedMapFile.MapFileWriter[]	forwardWriters
protected IndexOnDisk	index
protected DataOutputStream	indexOutput
protected gnu.trove.TObjectIntHashMap<String>	key2Index
protected int	keyCount
protected FixedSizeWriteableFactory<org.apache.hadoop.io.Text>[]	keyFactories
protected String[]	keyNames
protected String[]	lastValues
protected static org.slf4j.Logger	logger
protected static int	MAX_INDEX_MB_IN_MEM_RETRIEVAL
protected static int	MAX_MB_IN_MEM_RETRIEVAL
protected MemoryChecker	memCheck
protected static boolean	REVERSE_ALLOW_DUPS
protected static int	REVERSE_KEY_LOOKUP_WRITING_BUFFER_SIZE
protected byte[]	spaces
protected String	structureName
protected int[]	valueLensBytes
protected int[]	valueLensChars
protected Deflater	zip
protected static int	ZIP_COMPRESSION_LEVEL

Constructor Summary

Constructors

Constructor and Description
CompressingMetaIndexBuilder(IndexOnDisk _index, String[] _keyNames, int[] _valueLens, String[] _forwardKeys) constructor
CompressingMetaIndexBuilder(IndexOnDisk _index, String _structureName, String[] _keyNames, int[] _valueLens, String[] _forwardKeys) constructor

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	flush()
static void	main(String[] args)
static void	reverseAsMapReduceJob(IndexOnDisk index, String structureName, String[] keys) reverseAsMapReduceJob
static void	reverseAsMapReduceJob(IndexOnDisk index, String structureName, String[] keys, HadoopPlugin.JobFactory jf) reverseAsMapReduceJob
void	writeDocumentEntry(Map<String,String> data) Write out metadata for current document, extracted from specified map Typically, the MetaIndexBuilder will know which keys from data that it is interested in.
void	writeDocumentEntry(String[] data) Write out metadata for current document.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static final org.slf4j.Logger logger

MAX_MB_IN_MEM_RETRIEVAL

protected static final int MAX_MB_IN_MEM_RETRIEVAL

MAX_INDEX_MB_IN_MEM_RETRIEVAL

protected static final int MAX_INDEX_MB_IN_MEM_RETRIEVAL

REVERSE_ALLOW_DUPS

protected static final boolean REVERSE_ALLOW_DUPS

CROP_LONG

protected static final boolean CROP_LONG

REVERSE_KEY_LOOKUP_WRITING_BUFFER_SIZE

protected static final int REVERSE_KEY_LOOKUP_WRITING_BUFFER_SIZE

See Also:

Constant Field Values

DOCS_PER_CHECK

protected static final int DOCS_PER_CHECK

ZIP_COMPRESSION_LEVEL

protected static final int ZIP_COMPRESSION_LEVEL

See Also:

Constant Field Values

key2Index

protected final gnu.trove.TObjectIntHashMap<String> key2Index

dataOutput

protected DataOutputStream dataOutput

keyNames

protected final String[] keyNames

keyCount

protected final int keyCount

zip

protected Deflater zip

baos

protected ByteArrayOutputStream baos

indexOutput

protected DataOutputStream indexOutput

compressedBuffer

protected byte[] compressedBuffer

index

protected IndexOnDisk index

valueLensChars

protected int[] valueLensChars

valueLensBytes

protected int[] valueLensBytes

spaces

protected byte[] spaces

entryLengthBytes

protected int entryLengthBytes

currentOffset

protected long currentOffset

currentIndexOffset

protected long currentIndexOffset

entryCount

protected int entryCount

forwardKeys

protected int[] forwardKeys

forwardKeyNames

protected String[] forwardKeyNames

forwardWriters

protected FSOrderedMapFile.MapFileWriter[] forwardWriters

forwardKeyValuesSorted

protected boolean[] forwardKeyValuesSorted

lastValues

protected String[] lastValues

memCheck

protected MemoryChecker memCheck

keyFactories

protected FixedSizeWriteableFactory<org.apache.hadoop.io.Text>[] keyFactories

structureName

protected String structureName

Constructor Detail

CompressingMetaIndexBuilder

public CompressingMetaIndexBuilder(IndexOnDisk _index,
                           String[] _keyNames,
                           int[] _valueLens,
                           String[] _forwardKeys)

constructor

Parameters:

_index -

_keyNames -

_valueLens -

_forwardKeys -

CompressingMetaIndexBuilder

public CompressingMetaIndexBuilder(IndexOnDisk _index,
                           String _structureName,
                           String[] _keyNames,
                           int[] _valueLens,
                           String[] _forwardKeys)

constructor

Parameters:

_index -

_structureName -

_keyNames -

_valueLens -

_forwardKeys -

Method Detail

writeDocumentEntry

public void writeDocumentEntry(Map<String,String> data)
                        throws IOException

Write out metadata for current document, extracted from specified map Typically, the MetaIndexBuilder will know which keys from data that it is interested in.

Specified by:

writeDocumentEntry in class MetaIndexBuilder

Throws:

IOException

writeDocumentEntry

public void writeDocumentEntry(String[] data)
                        throws IOException

Write out metadata for current document. Values for all keys are specified.

Specified by:

writeDocumentEntry in class MetaIndexBuilder

Throws:

IOException

flush

public void flush()
           throws IOException

Specified by:

flush in interface Flushable

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

reverseAsMapReduceJob

public static void reverseAsMapReduceJob(IndexOnDisk index,
                         String structureName,
                         String[] keys)
                                  throws Exception

reverseAsMapReduceJob

Parameters:

index -

structureName -

keys -

Throws:

Exception

reverseAsMapReduceJob

public static void reverseAsMapReduceJob(IndexOnDisk index,
                         String structureName,
                         String[] keys,
                         HadoopPlugin.JobFactory jf)
                                  throws Exception

reverseAsMapReduceJob

Parameters:

index -

structureName -

keys -

jf -

Throws:

Exception