org.terrier.structures
Class Index

java.lang.Object
  extended by org.terrier.structures.Index
All Implemented Interfaces:
java.io.Closeable, java.io.Flushable

public class Index
extends java.lang.Object
implements java.io.Closeable, java.io.Flushable

This class encapsulates all the Indexes at retrieval time. It is loaded by giving a path and prefix. This looks for an index properties file at path/prefix.properties. Essentially, the properties file then specifies which index structures the index supports. The index then loads these so they can be used in retrieval.

Arbitrary properties can be defined in the index properties files, and in particular, properties are used to record index statistics and the contructor type and values of various index objects.

The Index will apply methods on specially marked interfaces. Currently, the only interface supported is IndexConfigurable. Moreover, structures implementing java.io.Closeable will have their close method called when the Index is closed.

Author:
Craig Macdonald & Vassilis Plachouras

Nested Class Summary
protected  class Index.UpdatingCollectionStatistics
          This collection statistics parses the associated index properties for each call.
 
Field Summary
protected  boolean dirtyProperties
          Have the properties of this index changed, suggesting a flush() is necessary when closing
protected static java.lang.Class<?>[] EMPTY_CLASS_ARRAY
          empty class array
protected static java.lang.String lastLoadError
           
protected  java.lang.String loadError
           
protected  boolean loadSuccess
          Set to true if loading an index succeeds
protected static org.apache.log4j.Logger logger
          The logger used
protected static int MINIMUM_INDEX_TERRIER_VERSION
           
protected  java.lang.String path
          path component of this index's location
protected  java.lang.String prefix
          prefix component of this index's location
protected  java.util.Properties properties
          properties of this index
protected static java.lang.String PROPERTIES_SUFFIX
           
protected static boolean RETRIEVAL_LOADING_PROFILE
           
protected  java.util.HashMap<java.lang.String,java.lang.Object> structureCache
          Cache of all opened index structures, but not input streams
 
Constructor Summary
protected Index()
          A default constuctor that creates an instance of the index.
protected Index(long a, long b, long c)
          A constructor for child classes that doesnt open the file
protected Index(java.lang.String _path, java.lang.String _prefix, boolean isNew)
          Constructs a new Index object.
 
Method Summary
 void addIndexStructure(java.lang.String structureName, java.lang.Class<?> className, java.lang.Class<?>[] paramTypes, java.lang.String[] paramValues)
          tell the index about a new index structure it provides.
 void addIndexStructure(java.lang.String structureName, java.lang.String className)
          add an index structure to this index.
 void addIndexStructure(java.lang.String structureName, java.lang.String className, java.lang.String[] paramTypes, java.lang.String[] paramValues)
          add an index structure to this index.
 void addIndexStructure(java.lang.String structureName, java.lang.String className, java.lang.String paramTypes, java.lang.String paramValues)
          add an index structure to this index.
 void addIndexStructureInputStream(java.lang.String structureName, java.lang.Class<?> className, java.lang.Class<?>[] paramTypes, java.lang.String[] paramValues)
          tell the index about a new input stream index structure it provides.
 void addIndexStructureInputStream(java.lang.String structureName, java.lang.String className)
          tell the index about a new input stream index structure it provides.
 void addIndexStructureInputStream(java.lang.String structureName, java.lang.String className, java.lang.String[] paramTypes, java.lang.String[] paramValues)
          tell the index about a new input stream index structure it provides.
 void addIndexStructureInputStream(java.lang.String structureName, java.lang.String className, java.lang.String paramTypes, java.lang.String paramValues)
          tell the index about a new input stream index structure it provides.
protected static boolean allExists(java.lang.String... files)
          returns true if all named files exist
 void close()
          Closes the data structures of the index.
static Index createIndex()
          Factory method for creating an index.
static Index createIndex(java.lang.String path, java.lang.String prefix)
          Factory method for load an index.
static Index createNewIndex(java.lang.String path, java.lang.String prefix)
          Factory method create a new index.
static boolean existsIndex(java.lang.String path, java.lang.String prefix)
          Returns true if it is likely that an index exists at the specified location
 void flush()
          Write any dirty data structures down to disk
protected  void flushProperties()
          Write any dirty properties down to disk
 CollectionStatistics getCollectionStatistics()
          Get the collection statistics
 DirectIndex getDirectIndex()
          Return the DirectIndex associated with this index
 DocumentIndex getDocumentIndex()
          Return the DocumentIndex associated with this index
static boolean getIndexLoadingProfileAsRetrieval()
          Get RETRIEVAL_LOADING_PROFILE
 java.lang.String getIndexProperty(java.lang.String key, java.lang.String defaultValue)
          get an arbitrary property in the index
 java.lang.Object getIndexStructure(java.lang.String structureName)
          Obtains the named index structure, using an already loaded one if possible.
 java.lang.Object getIndexStructureInputStream(java.lang.String structureName)
          Return the input stream associated with the specified structure of this index
 int getIntIndexProperty(java.lang.String key, int defaultValue)
          get an arbitrary int property from the index
 InvertedIndex getInvertedIndex()
          Returns the InvertedIndex to use for this index
static java.lang.String getLastIndexLoadError()
          Returns the last warning given by an index being loaded.
 Lexicon<java.lang.String> getLexicon()
          Return the Lexicon associated with this index
 MetaIndex getMetaIndex()
          Get the Meta Index structure
 java.lang.String getPath()
          Returns the path of this index
 java.lang.String getPrefix()
          Returns the prefix of this index
 java.util.Properties getProperties()
          Get the index properties
 boolean hasIndexStructure(java.lang.String structureName)
          Does this index have an index structure with the specified name?
 boolean hasIndexStructureInputStream(java.lang.String structureName)
          Does this index have an index structure input stream with the specified name?
protected static java.lang.String join(java.lang.String[] input, java.lang.String joinString)
          joins a series of strings together with a delimiter
protected  java.lang.Object loadIndexStructure(java.lang.String structureName)
          Load a new instance of the named index structure.
protected  void loadIndices()
          load all index structures.
protected  boolean loadProperties()
          loads in the properties file, falling back to the Terrier 1.xx log file if no properties exist.
protected  void loadStatistics()
          for an immutable index, use a normal collection statistics, never changes
protected  void loadUpdatingStatistics()
          for an index that is not yet built, use an UpdatingCollectionStatistics, which is slower but can support updates of the index statistics
static void main(java.lang.String[] args)
          main
static void setIndexLoadingProfileAsRetrieval(boolean yes)
          Set RETRIEVAL_LOADING_PROFILE
 void setIndexProperty(java.lang.String key, java.lang.String value)
          set an arbitrary property in the index
 java.lang.String toString()
          Returns a String representation of this index
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

MINIMUM_INDEX_TERRIER_VERSION

protected static final int MINIMUM_INDEX_TERRIER_VERSION
See Also:
Constant Field Values

PROPERTIES_SUFFIX

protected static final java.lang.String PROPERTIES_SUFFIX
See Also:
Constant Field Values

RETRIEVAL_LOADING_PROFILE

protected static boolean RETRIEVAL_LOADING_PROFILE

logger

protected static final org.apache.log4j.Logger logger
The logger used


lastLoadError

protected static java.lang.String lastLoadError

path

protected java.lang.String path
path component of this index's location


prefix

protected java.lang.String prefix
prefix component of this index's location


properties

protected final java.util.Properties properties
properties of this index


structureCache

protected final java.util.HashMap<java.lang.String,java.lang.Object> structureCache
Cache of all opened index structures, but not input streams


dirtyProperties

protected boolean dirtyProperties
Have the properties of this index changed, suggesting a flush() is necessary when closing


loadSuccess

protected boolean loadSuccess
Set to true if loading an index succeeds


loadError

protected java.lang.String loadError

EMPTY_CLASS_ARRAY

protected static final java.lang.Class<?>[] EMPTY_CLASS_ARRAY
empty class array

Constructor Detail

Index

protected Index(long a,
                long b,
                long c)
A constructor for child classes that doesnt open the file


Index

protected Index()
A default constuctor that creates an instance of the index.


Index

protected Index(java.lang.String _path,
                java.lang.String _prefix,
                boolean isNew)
Constructs a new Index object. Don't call this method, call the createIndex(String) factory method to construct an Index object.

Parameters:
_path - String the path in which the data structures will be created.
_prefix - String the prefix of the files to be created.
isNew - where a new Index should be created if there is no index at the specified location
Method Detail

setIndexLoadingProfileAsRetrieval

public static void setIndexLoadingProfileAsRetrieval(boolean yes)
Set RETRIEVAL_LOADING_PROFILE

Parameters:
yes -

getIndexLoadingProfileAsRetrieval

public static boolean getIndexLoadingProfileAsRetrieval()
Get RETRIEVAL_LOADING_PROFILE

Returns:
retrieval loading profile

getProperties

public java.util.Properties getProperties()
Get the index properties


loadStatistics

protected void loadStatistics()
for an immutable index, use a normal collection statistics, never changes


loadUpdatingStatistics

protected void loadUpdatingStatistics()
for an index that is not yet built, use an UpdatingCollectionStatistics, which is slower but can support updates of the index statistics


loadIndices

protected void loadIndices()
load all index structures. Is disabled if index property index.preloadIndices.disabled is set to true. It is false by default, which means that all non-inputstream indices are loaded on initialisation of the index. When the property is true, indices are loaded as required.


loadProperties

protected boolean loadProperties()
loads in the properties file, falling back to the Terrier 1.xx log file if no properties exist.


hasIndexStructure

public boolean hasIndexStructure(java.lang.String structureName)
Does this index have an index structure with the specified name?

Parameters:
structureName - name of the required structure
Returns:
true if the index has an appropriately named structure

hasIndexStructureInputStream

public boolean hasIndexStructureInputStream(java.lang.String structureName)
Does this index have an index structure input stream with the specified name?

Parameters:
structureName - name of the required structure
Returns:
true if the index has an appropriately named structure

getIndexStructure

public java.lang.Object getIndexStructure(java.lang.String structureName)
Obtains the named index structure, using an already loaded one if possible.

Parameters:
structureName - name of the required structure
Returns:
desired object or null if not found

loadIndexStructure

protected java.lang.Object loadIndexStructure(java.lang.String structureName)
Load a new instance of the named index structure.

Parameters:
structureName - name of the required structure
Returns:
desired object or null if not found

getIndexStructureInputStream

public java.lang.Object getIndexStructureInputStream(java.lang.String structureName)
Return the input stream associated with the specified structure of this index

Parameters:
structureName - The name of the structure of which you want the inputstream. Eg "lexicon"
Returns:
Required structure, or null if not found

close

public void close()
           throws java.io.IOException
Closes the data structures of the index.

Specified by:
close in interface java.io.Closeable
Throws:
java.io.IOException

flushProperties

protected void flushProperties()
                        throws java.io.IOException
Write any dirty properties down to disk

Throws:
java.io.IOException

flush

public void flush()
           throws java.io.IOException
Write any dirty data structures down to disk

Specified by:
flush in interface java.io.Flushable
Throws:
java.io.IOException

getInvertedIndex

public InvertedIndex getInvertedIndex()
Returns the InvertedIndex to use for this index


getDirectIndex

public DirectIndex getDirectIndex()
Return the DirectIndex associated with this index


getLexicon

public Lexicon<java.lang.String> getLexicon()
Return the Lexicon associated with this index


getDocumentIndex

public DocumentIndex getDocumentIndex()
Return the DocumentIndex associated with this index


getMetaIndex

public MetaIndex getMetaIndex()
Get the Meta Index structure


getCollectionStatistics

public CollectionStatistics getCollectionStatistics()
Get the collection statistics


toString

public java.lang.String toString()
Returns a String representation of this index

Overrides:
toString in class java.lang.Object

getPath

public java.lang.String getPath()
Returns the path of this index


getPrefix

public java.lang.String getPrefix()
Returns the prefix of this index


addIndexStructure

public void addIndexStructure(java.lang.String structureName,
                              java.lang.String className)
add an index structure to this index. Structure will be called structureName, and instantiated by a class called className. Instantiation parameters are "String,String", which are "path,prefix".

Parameters:
structureName -
className -

addIndexStructure

public void addIndexStructure(java.lang.String structureName,
                              java.lang.Class<?> className,
                              java.lang.Class<?>[] paramTypes,
                              java.lang.String[] paramValues)
tell the index about a new index structure it provides. Class and parameter types specified as Class objects instead of Strings.


addIndexStructure

public void addIndexStructure(java.lang.String structureName,
                              java.lang.String className,
                              java.lang.String[] paramTypes,
                              java.lang.String[] paramValues)
add an index structure to this index. Structure will be called structureName, and instantiated by a class called className. Instantiation type parameters or values are non-default.


addIndexStructure

public void addIndexStructure(java.lang.String structureName,
                              java.lang.String className,
                              java.lang.String paramTypes,
                              java.lang.String paramValues)
add an index structure to this index. Structure will be called structureName, and instantiated by a class called className. Instantiation type parameters or values are non-default.


addIndexStructureInputStream

public void addIndexStructureInputStream(java.lang.String structureName,
                                         java.lang.String className)
tell the index about a new input stream index structure it provides.


addIndexStructureInputStream

public void addIndexStructureInputStream(java.lang.String structureName,
                                         java.lang.Class<?> className,
                                         java.lang.Class<?>[] paramTypes,
                                         java.lang.String[] paramValues)
tell the index about a new input stream index structure it provides.


addIndexStructureInputStream

public void addIndexStructureInputStream(java.lang.String structureName,
                                         java.lang.String className,
                                         java.lang.String[] paramTypes,
                                         java.lang.String[] paramValues)
tell the index about a new input stream index structure it provides.


addIndexStructureInputStream

public void addIndexStructureInputStream(java.lang.String structureName,
                                         java.lang.String className,
                                         java.lang.String paramTypes,
                                         java.lang.String paramValues)
tell the index about a new input stream index structure it provides.


setIndexProperty

public void setIndexProperty(java.lang.String key,
                             java.lang.String value)
set an arbitrary property in the index

Parameters:
key - Key to of the property to set
value - Value of the property to set

getIndexProperty

public java.lang.String getIndexProperty(java.lang.String key,
                                         java.lang.String defaultValue)
get an arbitrary property in the index

Parameters:
key - Key of the property to get
defaultValue - value of the property to use if property is not set
Returns:
Value of the property

getIntIndexProperty

public int getIntIndexProperty(java.lang.String key,
                               int defaultValue)
get an arbitrary int property from the index


createIndex

public static Index createIndex(java.lang.String path,
                                java.lang.String prefix)
Factory method for load an index. This method should be used in order to load an existing index in the applications.

Parameters:
path - String the path in which the data structures will be created.
prefix - String the prefix of the files to be created.

createNewIndex

public static Index createNewIndex(java.lang.String path,
                                   java.lang.String prefix)
Factory method create a new index. This method should be used in order to load a new index in the applications.

Parameters:
path - String the path in which the data structures will be created.
prefix - String the prefix of the files to be created.

getLastIndexLoadError

public static java.lang.String getLastIndexLoadError()
Returns the last warning given by an index being loaded.


existsIndex

public static boolean existsIndex(java.lang.String path,
                                  java.lang.String prefix)
Returns true if it is likely that an index exists at the specified location

Parameters:
path -
prefix -
Returns:
true if a .properties or a .log files exists

createIndex

public static Index createIndex()
Factory method for creating an index. This method should be used in order to load an existing index in the applications.


allExists

protected static boolean allExists(java.lang.String... files)
returns true if all named files exist


join

protected static java.lang.String join(java.lang.String[] input,
                                       java.lang.String joinString)
joins a series of strings together with a delimiter


main

public static void main(java.lang.String[] args)
main

Parameters:
args -


Terrier 3.5. Copyright © 2004-2011 University of Glasgow