org.terrier.realtime.memory

Class MemoryIndex

java.lang.Object

org.terrier.structures.Index

org.terrier.realtime.memory.MemoryIndex

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable, UpdatableIndex, WritableIndex

Direct Known Subclasses:

MemoryFields

public class MemoryIndex
extends Index
implements UpdatableIndex, WritableIndex

An index held in fully memory. This is an updatable index, i.e. it supports indexDocument() methods which can be called at any time to add new documents. A MemoryIndex is also writable, i.e. it has a write() method that will convert it to an IndexOnDisk and write it out to the location specified by terrier.index.path and with prefix terrier.index.prefix.

Since:

4.0

Author:

Richard McCreadie, Dyaa Albakour

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MemoryIndex.Loader

Nested classes/interfaces inherited from class org.terrier.structures.Index

Index.UpdatingCollectionStatistics

Field Summary

Fields

Modifier and Type	Field and Description
static boolean	blocks
protected CompressionFactory.CompressionConfiguration	compressionConfig
protected MemoryDirectIndex	direct
protected DocumentPostingList	docPostings FIXME
protected MemoryDocumentIndex	document
static boolean	enableDirect
gnu.trove.TObjectIntHashMap<String>	fieldIDs
static boolean	fields
static String[]	fieldtags
protected Object	indexingLock A lock that stops multiple indexing operations from happening at once
protected MemoryInvertedIndex	inverted
protected MemoryLexicon	lexicon
protected static org.slf4j.Logger	logger
protected MemoryMetaIndex	metadata
protected TermPipeline	pipeline_first FIXME
protected static String	PIPELINE_NAMESPACE FIXME
protected MemoryCollectionStatistics	stats

Fields inherited from class org.terrier.structures.Index

dirtyProperties, EMPTY_CLASS_ARRAY, lastLoadError, MINIMUM_INDEX_TERRIER_VERSION, properties, PROPERTIES_SUFFIX, RETRIEVAL_LOADING_PROFILE

Constructor Summary

Constructors

Constructor and Description
MemoryIndex() Constructor.
MemoryIndex(IndexOnDisk superIndex)
MemoryIndex(IndexOnDisk superIndex, boolean compressedMeta) Create a memory index from an existing on-disk index.
MemoryIndex(MemoryLexicon tmplex, MemoryDocumentIndexMap document, MemoryInvertedIndex inverted, MemoryMetaIndex metadata, MemoryCollectionStatistics stats)

Method Summary

Modifier and Type	Method and Description
boolean	addToDocument(int docid, Document doc) Adds specified content contents to the named document id.
boolean	addToDocument(int docid, DocumentPostingList docContents) Adds relevant terms to the named document id.
void	close() Not implemented.
void	collectProperties(Index memory, Index newIndex, CompressionFactory.CompressionConfiguration compressionConfig) Collect index properties.
void	flush() Not implemented.
CollectionStatistics	getCollectionStatistics() Get the collection statistics
PostingIndex<?>	getDirectIndex() Not implemented.
DocumentIndex	getDocumentIndex() Return the DocumentIndex associated with this index
protected TermPipeline	getEndOfPipeline() FIXME
Object	getIndexStructure(String structureName) Obtains the named index structure, using an already loaded one if possible.
Object	getIndexStructureInputStream(String structureName) Return the input stream associated with the specified structure of this index
PostingIndex<?>	getInvertedIndex() Returns the InvertedIndex to use for this index
Lexicon<String>	getLexicon() Return the Lexicon associated with this index
MetaIndex	getMetaIndex() Get the Meta Index structure
void	indexDocument(Document doc) Index a new document.
void	indexDocument(Map<String,String> docProperties, DocumentPostingList docContents) Index a new document.
void	indexUnDocument(Document doc) Index an unsearchable document.
void	indexUnDocument(Map<String,String> docProperties, DocumentPostingList docContents) Index an unsearchable document.
protected void	load_pipeline() FIXME
boolean	removeDocument(int docid) Removes a document from the index.
String	toString() Returns a String representation of this index
Index	write(String path, String prefix) Write index structures to disk.

Methods inherited from class org.terrier.structures.Index

addIndexStructure, addIndexStructure, addIndexStructure, addIndexStructure, addIndexStructureInputStream, addIndexStructureInputStream, addIndexStructureInputStream, addIndexStructureInputStream, allExists, createIndex, createIndex, createNewIndex, existsIndex, getEnd, getIndexLoadingProfileAsRetrieval, getIndexProperty, getIndexRef, getIntIndexProperty, getLastIndexLoadError, getProperties, getStart, hasIndexStructure, hasIndexStructureInputStream, join, main, setIndexLoadingProfileAsRetrieval, setIndexProperty

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

logger

protected static final org.slf4j.Logger logger

lexicon

protected MemoryLexicon lexicon

inverted

protected MemoryInvertedIndex inverted

metadata

protected MemoryMetaIndex metadata

document

protected MemoryDocumentIndex document

stats

protected MemoryCollectionStatistics stats

direct

protected MemoryDirectIndex direct

blocks

public static final boolean blocks

fields

public static final boolean fields

fieldtags

public static final String[] fieldtags

fieldIDs

public gnu.trove.TObjectIntHashMap<String> fieldIDs

enableDirect

public static final boolean enableDirect

indexingLock

protected Object indexingLock

A lock that stops multiple indexing operations from happening at once

compressionConfig

protected CompressionFactory.CompressionConfiguration compressionConfig

docPostings

protected DocumentPostingList docPostings

FIXME

pipeline_first

protected TermPipeline pipeline_first

FIXME

PIPELINE_NAMESPACE

protected static final String PIPELINE_NAMESPACE

FIXME

See Also:

Constant Field Values

Constructor Detail

MemoryIndex

public MemoryIndex(MemoryLexicon tmplex,
                   MemoryDocumentIndexMap document,
                   MemoryInvertedIndex inverted,
                   MemoryMetaIndex metadata,
                   MemoryCollectionStatistics stats)

MemoryIndex

public MemoryIndex()

Constructor.

MemoryIndex

public MemoryIndex(IndexOnDisk superIndex)

MemoryIndex

public MemoryIndex(IndexOnDisk superIndex,
                   boolean compressedMeta)

Create a memory index from an existing on-disk index. If the restored index used non-incremental docis, the restoration will take that into account. This implementation will

Parameters:

superIndex -

compressedMeta -

Method Detail

getIndexStructure

public Object getIndexStructure(String structureName)

Obtains the named index structure, using an already loaded one if possible.

Specified by:

getIndexStructure in class Index

Parameters:

structureName - name of the required structure

Returns:

desired object or null if not found

getLexicon

public Lexicon<String> getLexicon()

Return the Lexicon associated with this index

Specified by:

getLexicon in class Index

getInvertedIndex

public PostingIndex<?> getInvertedIndex()

Returns the InvertedIndex to use for this index

Specified by:

getInvertedIndex in class Index

getMetaIndex

public MetaIndex getMetaIndex()

Get the Meta Index structure

Specified by:

getMetaIndex in class Index

getDocumentIndex

public DocumentIndex getDocumentIndex()

Return the DocumentIndex associated with this index

Specified by:

getDocumentIndex in class Index

getCollectionStatistics

public CollectionStatistics getCollectionStatistics()

Get the collection statistics

Specified by:

getCollectionStatistics in class Index

getDirectIndex

public PostingIndex<?> getDirectIndex()

Not implemented.

Specified by:

getDirectIndex in class Index

getIndexStructureInputStream

public Object getIndexStructureInputStream(String structureName)

Return the input stream associated with the specified structure of this index

Specified by:

getIndexStructureInputStream in class Index

Parameters:

structureName - The name of the structure of which you want the inputstream. Eg "lexicon"

Returns:

Required structure, or null if not found

indexDocument

public void indexDocument(Document doc)
                   throws Exception

Index a new document.

Specified by:

indexDocument in interface UpdatableIndex

Throws:

Exception

indexDocument

public void indexDocument(Map<String,String> docProperties,
                          DocumentPostingList docContents)
                   throws Exception

Index a new document.

Specified by:

indexDocument in interface UpdatableIndex

Throws:

Exception

indexUnDocument

public void indexUnDocument(Document doc)
                     throws Exception

Index an unsearchable document. This performs a partial index document operation. Term statistics and global statistics will be updated. But the inverted index will not be. In effect, this adds a document to the index, but in such a way that it will never be retrieved for any query. This is useful when you want to maintain collection statistics for a collection over time, but do not need search functionality.

Throws:

Exception

indexUnDocument

public void indexUnDocument(Map<String,String> docProperties,
                            DocumentPostingList docContents)
                     throws Exception

Index an unsearchable document. This performs a partial index document operation. Term statistics and global statistics will be updated. But the inverted index will not be. In effect, this adds a document to the index, but in such a way that it will never be retrieved for any query. This is useful when you want to maintain collection statistics for a collection over time, but do not need search functionality.

Throws:

Exception

addToDocument

public boolean addToDocument(int docid,
                             Document doc)
                      throws Exception

Adds specified content contents to the named document id.

NB: This implementation uses addToDocument(int, DocumentPostingList) internally.

Specified by:

addToDocument in interface UpdatableIndex

Returns:

true if supported & successful.

Throws:

Exception

addToDocument

public boolean addToDocument(int docid,
                             DocumentPostingList docContents)
                      throws Exception

Adds relevant terms to the named document id. Updates DocumentIndex and CollectionStatistics appropriately.

Specified by:

addToDocument in interface UpdatableIndex

Returns:

true if supported & successful.

Throws:

Exception

write

public Index write(String path,
                   String prefix)
            throws IOException

Write index structures to disk.

Specified by:

write in interface WritableIndex

Throws:

IOException

collectProperties

public void collectProperties(Index memory,
                              Index newIndex,
                              CompressionFactory.CompressionConfiguration compressionConfig)

Collect index properties.

Specified by:

collectProperties in interface WritableIndex

toString

public String toString()

Returns a String representation of this index

Specified by:

toString in class Index

close

public void close()
           throws IOException

Not implemented.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in class Index

Throws:

IOException

flush

public void flush()
           throws IOException

Not implemented.

Specified by:

flush in interface Flushable

Specified by:

flush in class Index

Throws:

IOException

getEndOfPipeline

protected TermPipeline getEndOfPipeline()

FIXME

load_pipeline

protected void load_pipeline()

FIXME

removeDocument

public boolean removeDocument(int docid)

Description copied from interface: UpdatableIndex

Removes a document from the index. Returns true if successful. No known operable implementations at this time.

Specified by:

removeDocument in interface UpdatableIndex