org.terrier.realtime.memory

Class MemoryCompressedMetaIndex

java.lang.Object

org.terrier.realtime.memory.MemoryMetaIndex

org.terrier.realtime.memory.MemoryCompressedMetaIndex

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, MetaIndexMap, MetaIndex

public class MemoryCompressedMetaIndex
extends MemoryMetaIndex
implements MetaIndexMap

An in-memory version of a Meta-data index. It stores additional information about each document, e.g. the docno or title. Access to the memory versions of the meta index are faster than the on-disk versions, but can use up large amounts of RAM to store. This is a Meta index that heavily compresses its content. This will make it slower to access but is useful when storing large volumes of content. It uses GZIP compression on each record in the meta index

Properties

• indexer.meta.forward.keys - key names to store in the meta index
• indexer.meta.forward.keylens - max key lengths for keys to store (this is ignored unless metaindex.crop is set)
• metaindex.crop - should the content to store be cropped down to the length specified in indexer.meta.forward.keylens?

Since:

4.0

Author:

Richard McCreadie, Dyaa Albakour

See Also:

Serialized Form

Field Summary

Fields inherited from class org.terrier.realtime.memory.MemoryMetaIndex

reverse

Constructor Summary

Constructors

Constructor and Description
MemoryCompressedMetaIndex() Constructor.
MemoryCompressedMetaIndex(String[] keys, int[] keylengths)

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Delete contents of metadata index (but keep keys).
String[]	getAllItems(int docid) Obtain all metadata for specified document.
int	getDocument(String key, String value) Not implemented.
String	getItem(String key, int docid) Obtain metadata of specified type for specified document.
String[]	getItems(String[] keys, int docid) Obtain metadata of specified types for specified document.
String[][]	getItems(String[] keys, int[] docids) Obtain metadata of specified types for specified documents.
String[]	getItems(String key, int[] docids) Obtain metadata of specified type for specified documents.
String[]	getKeys() Returns the keys of this meta index
Iterator<String[]>	iterator() Return iterator over meta-data index.
String[]	uncompress(byte[] compressed)
void	writeDocumentEntry(int docid, Map<String,String> data) Write meta-data for document.
void	writeDocumentEntry(int docid, String[] data)
void	writeDocumentEntry(String[] data) This method has been made public for creating

Methods inherited from class org.terrier.realtime.memory.MemoryMetaIndex

getKeyLengths, writeDocumentEntry

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MemoryCompressedMetaIndex

public MemoryCompressedMetaIndex(String[] keys,
                         int[] keylengths)

MemoryCompressedMetaIndex

public MemoryCompressedMetaIndex()

Constructor.

Method Detail

getKeys

public String[] getKeys()

Returns the keys of this meta index

Specified by:

getKeys in interface MetaIndex

Overrides:

getKeys in class MemoryMetaIndex

getItem

public String getItem(String key,
             int docid)
               throws IOException

Obtain metadata of specified type for specified document.

Specified by:

getItem in interface MetaIndex

Overrides:

getItem in class MemoryMetaIndex

Throws:

IOException

getAllItems

public String[] getAllItems(int docid)
                     throws IOException

Obtain all metadata for specified document.

Specified by:

getAllItems in interface MetaIndex

Overrides:

getAllItems in class MemoryMetaIndex

Throws:

IOException

getItems

public String[] getItems(String key,
                int[] docids)
                  throws IOException

Obtain metadata of specified type for specified documents.

Specified by:

getItems in interface MetaIndex

Overrides:

getItems in class MemoryMetaIndex

Throws:

IOException

getItems

public String[] getItems(String[] keys,
                int docid)
                  throws IOException

Obtain metadata of specified types for specified document.

Specified by:

getItems in interface MetaIndex

Overrides:

getItems in class MemoryMetaIndex

Throws:

IOException

getItems

public String[][] getItems(String[] keys,
                  int[] docids)
                    throws IOException

Obtain metadata of specified types for specified documents.

Specified by:

getItems in interface MetaIndex

Overrides:

getItems in class MemoryMetaIndex

Throws:

IOException

writeDocumentEntry

public void writeDocumentEntry(int docid,
                      Map<String,String> data)
                        throws Exception

Write meta-data for document.

Throws:

Exception

writeDocumentEntry

public void writeDocumentEntry(int docid,
                      String[] data)

Specified by:

writeDocumentEntry in interface MetaIndexMap

getDocument

public int getDocument(String key,
              String value)
                throws IOException

Not implemented.

Specified by:

getDocument in interface MetaIndex

Overrides:

getDocument in class MemoryMetaIndex

Throws:

IOException

close

public void close()
           throws IOException

Delete contents of metadata index (but keep keys).

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class MemoryMetaIndex

Throws:

IOException

writeDocumentEntry

public void writeDocumentEntry(String[] data)

Description copied from class: MemoryMetaIndex

This method has been made public for creating

Overrides:

writeDocumentEntry in class MemoryMetaIndex

iterator

public Iterator<String[]> iterator()

Return iterator over meta-data index.

Overrides:

iterator in class MemoryMetaIndex

uncompress

public String[] uncompress(byte[] compressed)