org.terrier.structures

Class CompressingMetaIndex

java.lang.Object

org.terrier.structures.CompressingMetaIndex

All Implemented Interfaces:

Closeable, AutoCloseable, MetaIndex

public class CompressingMetaIndex
extends Object
implements MetaIndex

A MetaIndex implementation that compresses contents. Values have maximum lengths, but overall value blobs are compressed using java.util.zip.Inflater.

Since:

3.0

Author:

Craig Macdonald & Vassilis Plachouras

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CompressingMetaIndex.CompressingMetaIndexInputFormat A Hadoop input format for a compressing meta index (allows the reading of a meta index as input to a MapReduce job.
static class	CompressingMetaIndex.InputStream An iterator for reading a MetaIndex as a stream

Field Summary

Fields

Modifier and Type	Field and Description
protected int	compressionLevel
protected org.terrier.structures.CompressingMetaIndex.ByteAccessor	dataSource
protected Map<org.apache.hadoop.io.Text,org.apache.hadoop.io.IntWritable>[]	forwardMetaMaps
protected static ThreadLocal<Inflater>	inflaterCache thread-local cache of Inflaters to be re-used for decompression
protected gnu.trove.TObjectIntHashMap<String>	key2bytelength
protected gnu.trove.TObjectIntHashMap<String>	key2byteoffset
protected gnu.trove.TObjectIntHashMap<String>	key2forwardOffset
protected int	keyCount
protected FixedSizeWriteableFactory<org.apache.hadoop.io.Text>[]	keyFactories
protected String[]	keyNames
protected org.terrier.structures.CompressingMetaIndex.Docid2OffsetLookup	offsetLookup
protected String	path
protected String	prefix
protected int	recordLength
protected int[]	valueByteLengths
protected int[]	valueByteOffsets

Constructor Summary

Constructors

Constructor and Description
CompressingMetaIndex(IndexOnDisk index, String structureName) Construct an instance of the class with

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Closes the underlying structures.
String[]	getAllItems(int docid) Obtain all metadata for specified document.
int	getDocument(String key, String value) Obtain docid where document has specified metadata value in the specified type.
String	getItem(String Key, int docid) Obtain metadata of specified type for specified document.
String[]	getItems(String[] Keys, int docid) Obtain metadata of specified types for specified document.
String[][]	getItems(String[] Keys, int[] _docids) Obtain metadata of specified types for specified documents.
String[]	getItems(String Key, int[] _docids) Obtain metadata of specified type for specified documents.
String[]	getKeys() Returns the keys of this meta index
protected void	loadIndex(IndexOnDisk index, String structureName)
static void	main(String[] args) main

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

inflaterCache

protected static final ThreadLocal<Inflater> inflaterCache

thread-local cache of Inflaters to be re-used for decompression

offsetLookup

protected org.terrier.structures.CompressingMetaIndex.Docid2OffsetLookup offsetLookup

compressionLevel

protected int compressionLevel

recordLength

protected int recordLength

keyNames

protected String[] keyNames

key2byteoffset

protected gnu.trove.TObjectIntHashMap<String> key2byteoffset

key2bytelength

protected gnu.trove.TObjectIntHashMap<String> key2bytelength

key2forwardOffset

protected gnu.trove.TObjectIntHashMap<String> key2forwardOffset

keyCount

protected int keyCount

valueByteOffsets

protected int[] valueByteOffsets

valueByteLengths

protected int[] valueByteLengths

path

protected final String path

prefix

protected final String prefix

dataSource

protected final org.terrier.structures.CompressingMetaIndex.ByteAccessor dataSource

forwardMetaMaps

protected Map<org.apache.hadoop.io.Text,org.apache.hadoop.io.IntWritable>[] forwardMetaMaps

keyFactories

protected FixedSizeWriteableFactory<org.apache.hadoop.io.Text>[] keyFactories

Constructor Detail

CompressingMetaIndex

public CompressingMetaIndex(IndexOnDisk index,
                    String structureName)
                     throws IOException

Construct an instance of the class with

Parameters:

index -

structureName -

Throws:

IOException

Method Detail

getKeys

public String[] getKeys()

Returns the keys of this meta index

Specified by:

getKeys in interface MetaIndex

close

public void close()
           throws IOException

Closes the underlying structures.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getDocument

public int getDocument(String key,
              String value)
                throws IOException

Obtain docid where document has specified metadata value in the specified type. Returns -1 if the value cannot be found for the specified key type.

Specified by:

getDocument in interface MetaIndex

Throws:

IOException

getItems

public String[] getItems(String Key,
                int[] _docids)
                  throws IOException

Obtain metadata of specified type for specified documents.. In this implementation, _docids are sorted to improve disk cache hits. _docids is however unchanged.

Specified by:

getItems in interface MetaIndex

Throws:

IOException

getItems

public String[][] getItems(String[] Keys,
                  int[] _docids)
                    throws IOException

Obtain metadata of specified types for specified documents. In this implementation, _docids are sorted to improve disk cache hits. _docids is however unchanged.

Specified by:

getItems in interface MetaIndex

Throws:

IOException

getItem

public String getItem(String Key,
             int docid)
               throws IOException

Obtain metadata of specified type for specified document.

Specified by:

getItem in interface MetaIndex

Throws:

IOException

getItems

public String[] getItems(String[] Keys,
                int docid)
                  throws IOException

Obtain metadata of specified types for specified document.

Specified by:

getItems in interface MetaIndex

Throws:

IOException

getAllItems

public String[] getAllItems(int docid)
                     throws IOException

Obtain all metadata for specified document.

Specified by:

getAllItems in interface MetaIndex

Throws:

IOException

loadIndex

protected void loadIndex(IndexOnDisk index,
             String structureName)
                  throws IOException

Throws:

IOException

main

public static void main(String[] args)
                 throws Exception

main

Parameters:

args -

Throws:

Exception