Package org.terrier.realtime.memory

Class MemoryCompressedMetaIndex

  • All Implemented Interfaces:
    java.io.Closeable, java.io.Serializable, java.lang.AutoCloseable, MetaIndexMap, MetaIndex
    public class MemoryCompressedMetaIndex
extends MemoryMetaIndex
implements MetaIndexMap
    An in-memory version of a Meta-data index. It stores additional information about each document, e.g. the docno or title. Access to the memory versions of the meta index are faster than the on-disk versions, but can use up large amounts of RAM to store. This is a Meta index that heavily compresses its content. This will make it slower to access but is useful when storing large volumes of content. It uses GZIP compression on each record in the meta index

    Properties

    • indexer.meta.forward.keys - key names to store in the meta index
    • indexer.meta.forward.keylens - max key lengths for keys to store (this is ignored unless metaindex.crop is set)
    • metaindex.compressed.crop.long - should the content to store be cropped down to the length specified in indexer.meta.forward.keylens?
    Since:
    4.0
    Author:
    Richard McCreadie, Dyaa Albakour
    See Also:
    Serialized Form

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()
      Delete contents of metadata index (but keep keys).
      java.lang.String[] getAllItems​(int docid)
      Obtain all metadata for specified document.
      int getDocument​(java.lang.String key, java.lang.String value)
      Not implemented.
      java.lang.String getItem​(java.lang.String key, int docid)
      Obtain metadata of specified type for specified document.
      java.lang.String[] getItems​(java.lang.String[] keys, int docid)
      Obtain metadata of specified types for specified document.
      java.lang.String[][] getItems​(java.lang.String[] keys, int[] docids)
      Obtain metadata of specified types for specified documents.
      java.lang.String[] getItems​(java.lang.String key, int[] docids)
      Obtain metadata of specified type for specified documents.
      java.lang.String[] getKeys()
      Returns the keys of this meta index
      java.util.Iterator<java.lang.String[]> iterator()
      Return iterator over meta-data index.
      java.lang.String[] uncompress​(byte[] compressed)  
      void writeDocumentEntry​(int docid, java.lang.String[] data)  
      void writeDocumentEntry​(int docid, java.util.Map<java.lang.String,​java.lang.String> data)
      Write meta-data for document.
      void writeDocumentEntry​(java.lang.String[] data)
      This method has been made public for creating

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • MemoryCompressedMetaIndex

        public MemoryCompressedMetaIndex​(java.lang.String[] keys,
                                 int[] keylengths)

      • MemoryCompressedMetaIndex

        public MemoryCompressedMetaIndex()
        Constructor.

    • Method Detail

      • getItem

        public java.lang.String getItem​(java.lang.String key,
                                int docid)
                         throws java.io.IOException
        Obtain metadata of specified type for specified document.
        Specified by:
        getItem in interface MetaIndex
        Overrides:
        getItem in class MemoryMetaIndex
        Throws:
        java.io.IOException

      • getAllItems

        public java.lang.String[] getAllItems​(int docid)
                               throws java.io.IOException
        Obtain all metadata for specified document.
        Specified by:
        getAllItems in interface MetaIndex
        Overrides:
        getAllItems in class MemoryMetaIndex
        Throws:
        java.io.IOException

      • getItems

        public java.lang.String[] getItems​(java.lang.String key,
                                   int[] docids)
                            throws java.io.IOException
        Obtain metadata of specified type for specified documents.
        Specified by:
        getItems in interface MetaIndex
        Overrides:
        getItems in class MemoryMetaIndex
        Throws:
        java.io.IOException

      • getItems

        public java.lang.String[] getItems​(java.lang.String[] keys,
                                   int docid)
                            throws java.io.IOException
        Obtain metadata of specified types for specified document.
        Specified by:
        getItems in interface MetaIndex
        Overrides:
        getItems in class MemoryMetaIndex
        Throws:
        java.io.IOException

      • getItems

        public java.lang.String[][] getItems​(java.lang.String[] keys,
                                     int[] docids)
                              throws java.io.IOException
        Obtain metadata of specified types for specified documents. Return array is indexed by documents, then by metakeys.
        Specified by:
        getItems in interface MetaIndex
        Overrides:
        getItems in class MemoryMetaIndex
        Throws:
        java.io.IOException

      • writeDocumentEntry

        public void writeDocumentEntry​(int docid,
                               java.util.Map<java.lang.String,​java.lang.String> data)
                        throws java.lang.Exception
        Write meta-data for document.
        Throws:
        java.lang.Exception

      • writeDocumentEntry

        public void writeDocumentEntry​(int docid,
                               java.lang.String[] data)
        Specified by:
        writeDocumentEntry in interface MetaIndexMap

      • getDocument

        public int getDocument​(java.lang.String key,
                       java.lang.String value)
                throws java.io.IOException
        Not implemented.
        Specified by:
        getDocument in interface MetaIndex
        Overrides:
        getDocument in class MemoryMetaIndex
        Throws:
        java.io.IOException

      • close

        public void close()
           throws java.io.IOException
        Delete contents of metadata index (but keep keys).
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Overrides:
        close in class MemoryMetaIndex
        Throws:
        java.io.IOException

      • iterator

        public java.util.Iterator<java.lang.String[]> iterator()
        Return iterator over meta-data index.
        Overrides:
        iterator in class MemoryMetaIndex

      • uncompress

        public java.lang.String[] uncompress​(byte[] compressed)