Package org.terrier.realtime.memory

Class MemoryIndex

java.lang.Object

org.terrier.structures.Index

org.terrier.realtime.memory.MemoryIndex

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, UpdatableIndex, WritableIndex

Direct Known Subclasses:

MemoryFields

public class MemoryIndex
extends Index
implements UpdatableIndex, WritableIndex

An index held in fully memory. This is an updatable index, i.e. it supports indexDocument() methods which can be called at any time to add new documents. A MemoryIndex is also writable, i.e. it has a write() method that will convert it to an IndexOnDisk and write it out to the location specified by terrier.index.path and with prefix terrier.index.prefix.

Since:

4.0

Author:

Richard McCreadie, Dyaa Albakour

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	MemoryIndex.Loader

Nested classes/interfaces inherited from class org.terrier.structures.Index

Index.DirectIndexRef

Field Summary

Fields

Modifier and Type	Field	Description
protected boolean	blocks
protected CompressionFactory.CompressionConfiguration	compressionDirectConfig
protected CompressionFactory.CompressionConfiguration	compressionInvertedConfig
protected MemoryDirectIndex	direct
protected DocumentPostingList	docPostings	FIXME
protected MemoryDocumentIndex	document
gnu.trove.TObjectIntHashMap<java.lang.String>	fieldIDs
protected boolean	fields
protected java.lang.String[]	fieldtags
protected java.lang.Object	indexingLock	A lock that stops multiple indexing operations from happening at once
protected MemoryInvertedIndex	inverted
protected MemoryLexicon	lexicon
protected static org.slf4j.Logger	logger
protected MemoryMetaIndex	metadata
protected TermPipeline	pipeline_first	FIXME
protected static java.lang.String	PIPELINE_NAMESPACE	FIXME
protected MemoryCollectionStatistics	stats

Constructor Summary

Constructors

Constructor	Description
MemoryIndex()	Constructor.
MemoryIndex(MemoryLexicon tmplex, MemoryDocumentIndexMap document, MemoryInvertedIndex inverted, MemoryMetaIndex metadata, MemoryCollectionStatistics stats)
MemoryIndex(IndexOnDisk superIndex)
MemoryIndex(IndexOnDisk superIndex, boolean compressedMeta)	Create a memory index from an existing on-disk index.

Method Summary

Modifier and Type	Method	Description
boolean	addToDocument(int docid, Document doc)	Adds specified content contents to the named document id.
boolean	addToDocument(int docid, DocumentPostingList docContents)	Adds relevant terms to the named document id.
void	close()	Not implemented.
void	flush()	Not implemented.
CollectionStatistics	getCollectionStatistics()	Get the collection statistics
PostingIndex<?>	getDirectIndex()	Not implemented.
DocumentIndex	getDocumentIndex()	Return the DocumentIndex associated with this index
protected TermPipeline	getEndOfPipeline()	FIXME
IndexRef	getIndexRef()	Returns a direct IndexRef to this index
java.lang.Object	getIndexStructure(java.lang.String structureName)	Return a particular index data structure
java.lang.Object	getIndexStructureInputStream(java.lang.String structureName)	Return a particular index data structure input stream
PostingIndex<?>	getInvertedIndex()	Returns the InvertedIndex to use for this index
Lexicon<java.lang.String>	getLexicon()	Return the Lexicon associated with this index
MetaIndex	getMetaIndex()	Get the Meta Index structure
boolean	hasIndexStructure(java.lang.String structureName)	Returns true iff the structure exists
void	indexDocument(java.util.Map<java.lang.String,java.lang.String> docProperties, DocumentPostingList docContents)	Index a new document.
void	indexDocument(Document doc)	Index a new document.
void	indexUnDocument(java.util.Map<java.lang.String,java.lang.String> docProperties, DocumentPostingList docContents)	Index an unsearchable document.
void	indexUnDocument(Document doc)	Index an unsearchable document.
protected void	load_pipeline()	FIXME
protected DiskIndexWriter	makeDiskIndexWriter(java.lang.String path, java.lang.String prefix)
boolean	removeDocument(int docid)	Removes a document from the index.
java.lang.String	toString()
Index	write(java.lang.String path, java.lang.String prefix)	Write index structures to disk.

Methods inherited from class org.terrier.structures.Index

createIndex, createIndex, getEnd, getStart, hasIndexStructureInputStream, makeDirectIndexRef, setIndexLoadingProfileAsRetrieval

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

logger

protected static final org.slf4j.Logger logger

lexicon

protected MemoryLexicon lexicon

inverted

protected MemoryInvertedIndex inverted

metadata

protected MemoryMetaIndex metadata

document

protected MemoryDocumentIndex document

stats

protected MemoryCollectionStatistics stats

direct

protected MemoryDirectIndex direct

blocks

protected boolean blocks

fields

protected boolean fields

fieldtags

protected java.lang.String[] fieldtags

fieldIDs

public gnu.trove.TObjectIntHashMap<java.lang.String> fieldIDs

indexingLock

protected java.lang.Object indexingLock

A lock that stops multiple indexing operations from happening at once

compressionInvertedConfig

protected CompressionFactory.CompressionConfiguration compressionInvertedConfig

compressionDirectConfig

protected CompressionFactory.CompressionConfiguration compressionDirectConfig

docPostings

protected DocumentPostingList docPostings

FIXME

pipeline_first

protected TermPipeline pipeline_first

FIXME

PIPELINE_NAMESPACE

protected static final java.lang.String PIPELINE_NAMESPACE

FIXME

See Also:

Constant Field Values

Constructor Detail

MemoryIndex

public MemoryIndex(MemoryLexicon tmplex,
                   MemoryDocumentIndexMap document,
                   MemoryInvertedIndex inverted,
                   MemoryMetaIndex metadata,
                   MemoryCollectionStatistics stats)

MemoryIndex

public MemoryIndex()

Constructor.

MemoryIndex

public MemoryIndex(IndexOnDisk superIndex)

MemoryIndex

public MemoryIndex(IndexOnDisk superIndex,
                   boolean compressedMeta)

Create a memory index from an existing on-disk index. If the restored index used non-incremental docis, the restoration will take that into account. This implementation will

Parameters:

superIndex -

compressedMeta -

Method Detail

getIndexStructure

public java.lang.Object getIndexStructure(java.lang.String structureName)

Return a particular index data structure

Overrides:

getIndexStructure in class Index

getIndexStructureInputStream

public java.lang.Object getIndexStructureInputStream(java.lang.String structureName)

Description copied from class: Index

Return a particular index data structure input stream

Overrides:

getIndexStructureInputStream in class Index

getLexicon

public Lexicon<java.lang.String> getLexicon()

Return the Lexicon associated with this index

Specified by:

getLexicon in class Index

getInvertedIndex

public PostingIndex<?> getInvertedIndex()

Returns the InvertedIndex to use for this index

Specified by:

getInvertedIndex in class Index

getMetaIndex

public MetaIndex getMetaIndex()

Get the Meta Index structure

Specified by:

getMetaIndex in class Index

getDocumentIndex

public DocumentIndex getDocumentIndex()

Return the DocumentIndex associated with this index

Specified by:

getDocumentIndex in class Index

getCollectionStatistics

public CollectionStatistics getCollectionStatistics()

Get the collection statistics

Specified by:

getCollectionStatistics in class Index

getDirectIndex

public PostingIndex<?> getDirectIndex()

Not implemented.

Specified by:

getDirectIndex in class Index

getIndexRef

public IndexRef getIndexRef()

Description copied from class: Index

Returns a direct IndexRef to this index

Specified by:

getIndexRef in class Index

indexDocument

public void indexDocument(Document doc)
                   throws java.lang.Exception

Index a new document.

Specified by:

indexDocument in interface UpdatableIndex

Throws:

java.lang.Exception

hasIndexStructure

public boolean hasIndexStructure(java.lang.String structureName)

Description copied from class: Index

Returns true iff the structure exists

Overrides:

hasIndexStructure in class Index

indexDocument

public void indexDocument(java.util.Map<java.lang.String,java.lang.String> docProperties,
                          DocumentPostingList docContents)
                   throws java.lang.Exception

Index a new document.

Specified by:

indexDocument in interface UpdatableIndex

Throws:

java.lang.Exception

indexUnDocument

public void indexUnDocument(Document doc)
                     throws java.lang.Exception

Index an unsearchable document. This performs a partial index document operation. Term statistics and global statistics will be updated. But the inverted index will not be. In effect, this adds a document to the index, but in such a way that it will never be retrieved for any query. This is useful when you want to maintain collection statistics for a collection over time, but do not need search functionality.

Throws:

java.lang.Exception

indexUnDocument

public void indexUnDocument(java.util.Map<java.lang.String,java.lang.String> docProperties,
                            DocumentPostingList docContents)
                     throws java.lang.Exception

Index an unsearchable document. This performs a partial index document operation. Term statistics and global statistics will be updated. But the inverted index will not be. In effect, this adds a document to the index, but in such a way that it will never be retrieved for any query. This is useful when you want to maintain collection statistics for a collection over time, but do not need search functionality.

Throws:

java.lang.Exception

addToDocument

public boolean addToDocument(int docid,
                             Document doc)
                      throws java.lang.Exception

Adds specified content contents to the named document id.

NB: This implementation uses addToDocument(int, DocumentPostingList) internally.

Specified by:

addToDocument in interface UpdatableIndex

Returns:

true if supported & successful.

Throws:

java.lang.Exception

addToDocument

public boolean addToDocument(int docid,
                             DocumentPostingList docContents)
                      throws java.lang.Exception

Adds relevant terms to the named document id. Updates DocumentIndex and CollectionStatistics appropriately.

Specified by:

addToDocument in interface UpdatableIndex

Returns:

true if supported & successful.

Throws:

java.lang.Exception

write

public Index write(java.lang.String path,
                   java.lang.String prefix)
            throws java.io.IOException

Write index structures to disk.

Specified by:

write in interface WritableIndex

Throws:

java.io.IOException

makeDiskIndexWriter

protected DiskIndexWriter makeDiskIndexWriter(java.lang.String path,
                                              java.lang.String prefix)

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

close

public void close()
           throws java.io.IOException

Not implemented.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Overrides:

close in class Index

Throws:

java.io.IOException

flush

public void flush()
           throws java.io.IOException

Not implemented.

Throws:

java.io.IOException

getEndOfPipeline

protected TermPipeline getEndOfPipeline()

FIXME

load_pipeline

protected void load_pipeline()

FIXME

removeDocument

public boolean removeDocument(int docid)

Description copied from interface: UpdatableIndex

Removes a document from the index. Returns true if successful. No known operable implementations at this time.

Specified by:

removeDocument in interface UpdatableIndex