org.terrier.realtime.incremental

Class IncrementalIndex

java.lang.Object

org.terrier.structures.Index

org.terrier.realtime.multi.MultiIndex

org.terrier.realtime.incremental.IncrementalIndex

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable, UpdatableIndex

public class IncrementalIndex
extends MultiIndex
implements UpdatableIndex

This is the main Index class for an incremental index. An incremental index is a MultiIndex where one index shard is stored in memory and the rest are stored on disk. Periodically, the memory index is then written do disk, defined as per a FlushPolicy. When the memory index has been flushed to disk, optionally the on-disk portion of the incremental index can then be merged together (based upon a MergePolicy) and/or deleted (based upon a DeletePolicy).

Properties

• incremental.flush: the flush policy to use. Four possible values are supported: noflush (default), flushdocs, flushmem, flushtime

• incremental.merge: the merge policy to use. Three possible values are supported: nomerge (default), single, geometric

• incremental.delete: the delete policy to use. Two possible values are supported: nodelete (default), deleteFixedSize

Since:

4.0

Author:

Richard McCreadie, Stuart Mackie

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IncrementalIndex.Loader

Nested classes/interfaces inherited from class org.terrier.structures.Index

Index.UpdatingCollectionStatistics

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	logger
MemoryIndex	memory
String	path
String	prefix
int	prefixID

Fields inherited from class org.terrier.realtime.multi.MultiIndex

indices, selectiveMatchingPolicy

Fields inherited from class org.terrier.structures.Index

dirtyProperties, EMPTY_CLASS_ARRAY, lastLoadError, MINIMUM_INDEX_TERRIER_VERSION, properties, PROPERTIES_SUFFIX, RETRIEVAL_LOADING_PROFILE

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	IncrementalIndex(Index[] indices)

Method Summary

Modifier and Type	Method and Description
boolean	addToDocument(int docid, Document doc) Adds specified content contents to the named document id.
boolean	addToDocument(int docid, DocumentPostingList docContents) Adds relevant terms to the named document id.
void	close() Closes the data structures of the index.
void	flush() Write any dirty data structures down to disk
static IncrementalIndex	get(String path, String prefix) Construct a new incremental index.
String	getPath()
String	getPrefix()
int	getPrefixID()
String	getTimeOfLastUpdate() This method prints out the last time this index was updated as a String in GMT format
void	indexDocument(Document doc) Update the index with a new document.
void	indexDocument(Map<String,String> docProperties, DocumentPostingList docContents) Update the index with a new document.
boolean	removeDocument(int docid) Removes a document from the index.
void	setPath(String path)
void	setPrefix(String prefix)
void	setPrefixID(int prefixID)

Methods inherited from class org.terrier.realtime.multi.MultiIndex

getCollectionStatistics, getDirectIndex, getDocumentIndex, getIndexStructure, getIndexStructureInputStream, getInvertedIndex, getIthShard, getLexicon, getMetaIndex, getNumberOfShards, toString

Methods inherited from class org.terrier.structures.Index

addIndexStructure, addIndexStructure, addIndexStructure, addIndexStructure, addIndexStructureInputStream, addIndexStructureInputStream, addIndexStructureInputStream, addIndexStructureInputStream, allExists, createIndex, createIndex, createNewIndex, existsIndex, getEnd, getIndexLoadingProfileAsRetrieval, getIndexProperty, getIndexRef, getIntIndexProperty, getLastIndexLoadError, getProperties, getStart, hasIndexStructure, hasIndexStructureInputStream, join, main, setIndexLoadingProfileAsRetrieval, setIndexProperty

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

logger

protected static final org.slf4j.Logger logger

memory

public MemoryIndex memory

path

public String path

prefix

public String prefix

prefixID

public int prefixID

Constructor Detail

IncrementalIndex

protected IncrementalIndex(Index[] indices)

Method Detail

get

public static IncrementalIndex get(String path,
                                   String prefix)

Construct a new incremental index.

indexDocument

public void indexDocument(Document doc)
                   throws Exception

Update the index with a new document.

Specified by:

indexDocument in interface UpdatableIndex

Throws:

Exception

indexDocument

public void indexDocument(Map<String,String> docProperties,
                          DocumentPostingList docContents)
                   throws Exception

Update the index with a new document.

Specified by:

indexDocument in interface UpdatableIndex

Throws:

Exception

flush

public void flush()
           throws IOException

Write any dirty data structures down to disk

Specified by:

flush in interface Flushable

Overrides:

flush in class MultiIndex

Throws:

IOException

close

public void close()
           throws IOException

Closes the data structures of the index.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class MultiIndex

Throws:

IOException

getTimeOfLastUpdate

public String getTimeOfLastUpdate()

This method prints out the last time this index was updated as a String in GMT format

getPath

public String getPath()

setPath

public void setPath(String path)

getPrefix

public String getPrefix()

setPrefix

public void setPrefix(String prefix)

getPrefixID

public int getPrefixID()

setPrefixID

public void setPrefixID(int prefixID)

removeDocument

public boolean removeDocument(int docid)

Description copied from interface: UpdatableIndex

Removes a document from the index. Returns true if successful. No known operable implementations at this time.

Specified by:

removeDocument in interface UpdatableIndex

addToDocument

public boolean addToDocument(int docid,
                             Document doc)
                      throws Exception

Description copied from interface: UpdatableIndex

Adds specified content contents to the named document id.

Specified by:

addToDocument in interface UpdatableIndex

Returns:

true if supported & successful.

Throws:

Exception

addToDocument

public boolean addToDocument(int docid,
                             DocumentPostingList docContents)
                      throws Exception

Description copied from interface: UpdatableIndex

Adds relevant terms to the named document id.

Specified by:

addToDocument in interface UpdatableIndex

Returns:

true if supported & successful.

Throws:

Exception