org.terrier.structures
Class Index

java.lang.Object
  org.terrier.structures.Index

All Implemented Interfaces:

java.io.Closeable, java.io.Flushable

public class Index
extends java.lang.Object
implements java.io.Closeable, java.io.Flushable

This class encapsulates all the Indexes at retrieval time. It is loaded by giving a path and prefix. This looks for an index properties file at path/prefix.properties. Essentially, the properties file then specifies which index structures the index supports. The index then loads these so they can be used in retrieval.

Arbitrary properties can be defined in the index properties files, and in particular, properties are used to record index statistics and the contructor type and values of various index objects.

The Index will apply methods on specially marked interfaces. Currently, the only interface supported is IndexConfigurable. Moreover, structures implementing java.io.Closeable will have their close method called when the Index is closed.

Author:

Craig Macdonald & Vassilis Plachouras

Nested Class Summary

protected class

Index.UpdatingCollectionStatistics This collection statistics parses the associated index properties for each call.

Field Summary

protected boolean	dirtyProperties Have the properties of this index changed, suggesting a flush() is necessary when closing
protected static java.lang.Class<?>[]	EMPTY_CLASS_ARRAY empty class array
protected static java.lang.String	lastLoadError
protected java.lang.String	loadError
protected boolean	loadSuccess Set to true if loading an index succeeds
protected static org.apache.log4j.Logger	logger The logger used
protected static int	MINIMUM_INDEX_TERRIER_VERSION
protected java.lang.String	path path component of this index's location
protected java.lang.String	prefix prefix component of this index's location
protected java.util.Properties	properties properties of this index
protected static java.lang.String	PROPERTIES_SUFFIX
protected static boolean	RETRIEVAL_LOADING_PROFILE
protected java.util.HashMap<java.lang.String,java.lang.Object>	structureCache Cache of all opened index structures, but not input streams

Constructor Summary

protected	Index() A default constuctor that creates an instance of the index.
protected	Index(long a, long b, long c) A constructor for child classes that doesnt open the file
protected	Index(java.lang.String _path, java.lang.String _prefix, boolean isNew) Constructs a new Index object.

Method Summary

void	addIndexStructure(java.lang.String structureName, java.lang.Class<?> className, java.lang.Class<?>[] paramTypes, java.lang.String[] paramValues) tell the index about a new index structure it provides.
void	addIndexStructure(java.lang.String structureName, java.lang.String className) add an index structure to this index.
void	addIndexStructure(java.lang.String structureName, java.lang.String className, java.lang.String[] paramTypes, java.lang.String[] paramValues) add an index structure to this index.
void	addIndexStructure(java.lang.String structureName, java.lang.String className, java.lang.String paramTypes, java.lang.String paramValues) add an index structure to this index.
void	addIndexStructureInputStream(java.lang.String structureName, java.lang.Class<?> className, java.lang.Class<?>[] paramTypes, java.lang.String[] paramValues) tell the index about a new input stream index structure it provides.
void	addIndexStructureInputStream(java.lang.String structureName, java.lang.String className) tell the index about a new input stream index structure it provides.
void	addIndexStructureInputStream(java.lang.String structureName, java.lang.String className, java.lang.String[] paramTypes, java.lang.String[] paramValues) tell the index about a new input stream index structure it provides.
void	addIndexStructureInputStream(java.lang.String structureName, java.lang.String className, java.lang.String paramTypes, java.lang.String paramValues) tell the index about a new input stream index structure it provides.
protected static boolean	allExists(java.lang.String... files) returns true if all named files exist
void	close() Closes the data structures of the index.
static Index	createIndex() Factory method for creating an index.
static Index	createIndex(java.lang.String path, java.lang.String prefix) Factory method for load an index.
static Index	createNewIndex(java.lang.String path, java.lang.String prefix) Factory method create a new index.
static boolean	existsIndex(java.lang.String path, java.lang.String prefix) Returns true if it is likely that an index exists at the specified location
void	flush() Write any dirty data structures down to disk
protected void	flushProperties() Write any dirty properties down to disk
CollectionStatistics	getCollectionStatistics() Get the collection statistics
DirectIndex	getDirectIndex() Return the DirectIndex associated with this index
DocumentIndex	getDocumentIndex() Return the DocumentIndex associated with this index
static boolean	getIndexLoadingProfileAsRetrieval() Get RETRIEVAL_LOADING_PROFILE
java.lang.String	getIndexProperty(java.lang.String key, java.lang.String defaultValue) get an arbitrary property in the index
java.lang.Object	getIndexStructure(java.lang.String structureName) Obtains the named index structure, using an already loaded one if possible.
java.lang.Object	getIndexStructureInputStream(java.lang.String structureName) Return the input stream associated with the specified structure of this index
int	getIntIndexProperty(java.lang.String key, int defaultValue) get an arbitrary int property from the index
InvertedIndex	getInvertedIndex() Returns the InvertedIndex to use for this index
static java.lang.String	getLastIndexLoadError() Returns the last warning given by an index being loaded.
Lexicon<java.lang.String>	getLexicon() Return the Lexicon associated with this index
MetaIndex	getMetaIndex() Get the Meta Index structure
java.lang.String	getPath() Returns the path of this index
java.lang.String	getPrefix() Returns the prefix of this index
java.util.Properties	getProperties() Get the index properties
boolean	hasIndexStructure(java.lang.String structureName) Does this index have an index structure with the specified name?
boolean	hasIndexStructureInputStream(java.lang.String structureName) Does this index have an index structure input stream with the specified name?
protected static java.lang.String	join(java.lang.String[] input, java.lang.String joinString) joins a series of strings together with a delimiter
protected java.lang.Object	loadIndexStructure(java.lang.String structureName) Load a new instance of the named index structure.
protected void	loadIndices() load all index structures.
protected boolean	loadProperties() loads in the properties file, falling back to the Terrier 1.xx log file if no properties exist.
protected void	loadStatistics() for an immutable index, use a normal collection statistics, never changes
protected void	loadUpdatingStatistics() for an index that is not yet built, use an UpdatingCollectionStatistics, which is slower but can support updates of the index statistics
static void	main(java.lang.String[] args) main
static void	setIndexLoadingProfileAsRetrieval(boolean yes) Set RETRIEVAL_LOADING_PROFILE
void	setIndexProperty(java.lang.String key, java.lang.String value) set an arbitrary property in the index
java.lang.String	toString() Returns a String representation of this index

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

MINIMUM_INDEX_TERRIER_VERSION

protected static final int MINIMUM_INDEX_TERRIER_VERSION

See Also:

Constant Field Values

PROPERTIES_SUFFIX

protected static final java.lang.String PROPERTIES_SUFFIX

See Also:

Constant Field Values

RETRIEVAL_LOADING_PROFILE

protected static boolean RETRIEVAL_LOADING_PROFILE

logger

protected static final org.apache.log4j.Logger logger

The logger used

lastLoadError

protected static java.lang.String lastLoadError

path

protected java.lang.String path

path component of this index's location

prefix

protected java.lang.String prefix

prefix component of this index's location

properties

protected final java.util.Properties properties

properties of this index

structureCache

protected final java.util.HashMap<java.lang.String,java.lang.Object> structureCache

Cache of all opened index structures, but not input streams

dirtyProperties

protected boolean dirtyProperties

Have the properties of this index changed, suggesting a flush() is necessary when closing

loadSuccess

protected boolean loadSuccess

Set to true if loading an index succeeds

loadError

protected java.lang.String loadError

EMPTY_CLASS_ARRAY

protected static final java.lang.Class<?>[] EMPTY_CLASS_ARRAY

empty class array

Constructor Detail

Index

protected Index(long a,
                long b,
                long c)

A constructor for child classes that doesnt open the file

Index

protected Index()

A default constuctor that creates an instance of the index.

Index

protected Index(java.lang.String _path,
                java.lang.String _prefix,
                boolean isNew)

Constructs a new Index object. Don't call this method, call the createIndex(String) factory method to construct an Index object.

Parameters:

_path - String the path in which the data structures will be created.

_prefix - String the prefix of the files to be created.

isNew - where a new Index should be created if there is no index at the specified location

Method Detail

setIndexLoadingProfileAsRetrieval

public static void setIndexLoadingProfileAsRetrieval(boolean yes)

Set RETRIEVAL_LOADING_PROFILE

Parameters:

yes -

getIndexLoadingProfileAsRetrieval

public static boolean getIndexLoadingProfileAsRetrieval()

Get RETRIEVAL_LOADING_PROFILE

Returns:

retrieval loading profile

getProperties

public java.util.Properties getProperties()

Get the index properties

loadStatistics

protected void loadStatistics()

for an immutable index, use a normal collection statistics, never changes

loadUpdatingStatistics

protected void loadUpdatingStatistics()

for an index that is not yet built, use an UpdatingCollectionStatistics, which is slower but can support updates of the index statistics

loadIndices

protected void loadIndices()

load all index structures. Is disabled if index property index.preloadIndices.disabled is set to true. It is false by default, which means that all non-inputstream indices are loaded on initialisation of the index. When the property is true, indices are loaded as required.

loadProperties

protected boolean loadProperties()

loads in the properties file, falling back to the Terrier 1.xx log file if no properties exist.

hasIndexStructure

public boolean hasIndexStructure(java.lang.String structureName)

Does this index have an index structure with the specified name?

Parameters:

structureName - name of the required structure

Returns:

true if the index has an appropriately named structure

hasIndexStructureInputStream

public boolean hasIndexStructureInputStream(java.lang.String structureName)

Does this index have an index structure input stream with the specified name?

Parameters:

structureName - name of the required structure

Returns:

true if the index has an appropriately named structure

getIndexStructure

public java.lang.Object getIndexStructure(java.lang.String structureName)

Obtains the named index structure, using an already loaded one if possible.

Parameters:

structureName - name of the required structure

Returns:

desired object or null if not found

loadIndexStructure

protected java.lang.Object loadIndexStructure(java.lang.String structureName)

Load a new instance of the named index structure.

Parameters:

structureName - name of the required structure

Returns:

desired object or null if not found

getIndexStructureInputStream

public java.lang.Object getIndexStructureInputStream(java.lang.String structureName)

Return the input stream associated with the specified structure of this index

Parameters:

structureName - The name of the structure of which you want the inputstream. Eg "lexicon"

Returns:

Required structure, or null if not found

close

public void close()
           throws java.io.IOException

Closes the data structures of the index.

Specified by:

close in interface java.io.Closeable

Throws:

java.io.IOException

flushProperties

protected void flushProperties()
                        throws java.io.IOException

Write any dirty properties down to disk

Throws:

java.io.IOException

flush

public void flush()
           throws java.io.IOException

Write any dirty data structures down to disk

Specified by:

flush in interface java.io.Flushable

Throws:

java.io.IOException

getInvertedIndex

public InvertedIndex getInvertedIndex()

Returns the InvertedIndex to use for this index

getDirectIndex

public DirectIndex getDirectIndex()

Return the DirectIndex associated with this index

getLexicon

public Lexicon<java.lang.String> getLexicon()

Return the Lexicon associated with this index

getDocumentIndex

public DocumentIndex getDocumentIndex()

Return the DocumentIndex associated with this index

getMetaIndex

public MetaIndex getMetaIndex()

Get the Meta Index structure

getCollectionStatistics

public CollectionStatistics getCollectionStatistics()

Get the collection statistics

toString

public java.lang.String toString()

Returns a String representation of this index

Overrides:

toString in class java.lang.Object

getPath

public java.lang.String getPath()

Returns the path of this index

getPrefix

public java.lang.String getPrefix()

Returns the prefix of this index

addIndexStructure

public void addIndexStructure(java.lang.String structureName,
                              java.lang.String className)

add an index structure to this index. Structure will be called structureName, and instantiated by a class called className. Instantiation parameters are "String,String", which are "path,prefix".

Parameters:

structureName -

className -

addIndexStructure

public void addIndexStructure(java.lang.String structureName,
                              java.lang.Class<?> className,
                              java.lang.Class<?>[] paramTypes,
                              java.lang.String[] paramValues)

tell the index about a new index structure it provides. Class and parameter types specified as Class objects instead of Strings.

addIndexStructure

public void addIndexStructure(java.lang.String structureName,
                              java.lang.String className,
                              java.lang.String[] paramTypes,
                              java.lang.String[] paramValues)

add an index structure to this index. Structure will be called structureName, and instantiated by a class called className. Instantiation type parameters or values are non-default.

addIndexStructure

public void addIndexStructure(java.lang.String structureName,
                              java.lang.String className,
                              java.lang.String paramTypes,
                              java.lang.String paramValues)

add an index structure to this index. Structure will be called structureName, and instantiated by a class called className. Instantiation type parameters or values are non-default.

addIndexStructureInputStream

public void addIndexStructureInputStream(java.lang.String structureName,
                                         java.lang.String className)

tell the index about a new input stream index structure it provides.

addIndexStructureInputStream

public void addIndexStructureInputStream(java.lang.String structureName,
                                         java.lang.Class<?> className,
                                         java.lang.Class<?>[] paramTypes,
                                         java.lang.String[] paramValues)

tell the index about a new input stream index structure it provides.

addIndexStructureInputStream

public void addIndexStructureInputStream(java.lang.String structureName,
                                         java.lang.String className,
                                         java.lang.String[] paramTypes,
                                         java.lang.String[] paramValues)

tell the index about a new input stream index structure it provides.

addIndexStructureInputStream

public void addIndexStructureInputStream(java.lang.String structureName,
                                         java.lang.String className,
                                         java.lang.String paramTypes,
                                         java.lang.String paramValues)

tell the index about a new input stream index structure it provides.

setIndexProperty

public void setIndexProperty(java.lang.String key,
                             java.lang.String value)

set an arbitrary property in the index

Parameters:

key - Key to of the property to set

value - Value of the property to set

getIndexProperty

public java.lang.String getIndexProperty(java.lang.String key,
                                         java.lang.String defaultValue)

get an arbitrary property in the index

Parameters:

key - Key of the property to get

defaultValue - value of the property to use if property is not set

Returns:

Value of the property

getIntIndexProperty

public int getIntIndexProperty(java.lang.String key,
                               int defaultValue)

get an arbitrary int property from the index

createIndex

public static Index createIndex(java.lang.String path,
                                java.lang.String prefix)

Factory method for load an index. This method should be used in order to load an existing index in the applications.

Parameters:

path - String the path in which the data structures will be created.

prefix - String the prefix of the files to be created.

createNewIndex

public static Index createNewIndex(java.lang.String path,
                                   java.lang.String prefix)

Factory method create a new index. This method should be used in order to load a new index in the applications.

Parameters:

path - String the path in which the data structures will be created.

prefix - String the prefix of the files to be created.

getLastIndexLoadError

public static java.lang.String getLastIndexLoadError()

Returns the last warning given by an index being loaded.

existsIndex

public static boolean existsIndex(java.lang.String path,
                                  java.lang.String prefix)

Returns true if it is likely that an index exists at the specified location

Parameters:

path -

prefix -

Returns:

true if a .properties or a .log files exists

createIndex

public static Index createIndex()

Factory method for creating an index. This method should be used in order to load an existing index in the applications.

allExists

protected static boolean allExists(java.lang.String... files)

returns true if all named files exist

join

protected static java.lang.String join(java.lang.String[] input,
                                       java.lang.String joinString)

joins a series of strings together with a delimiter

main

public static void main(java.lang.String[] args)

main

Parameters:

args -