org.terrier.structures

Class FSOMapFileLexicon

java.lang.Object

org.terrier.structures.Lexicon<String>

org.terrier.structures.MapLexicon

org.terrier.structures.FSOMapFileLexicon

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<Map.Entry<String,LexiconEntry>>

public class FSOMapFileLexicon
extends MapLexicon

Instance of a Lexicon where a FSOrderedMapFile is always used as a backing store.

Index Properties:

• index.STRUCTURENAME.termids - one of {aligned, file, fileinmem, disabled}. Depicts how termid to term lookups are handled.
• index.STRUCTURENAME.data-source - one of {file, fileinmem}. Depicts how the lexicon will be read. file means on disk, fileinmem means the entire lexicon file will be read into memory and read from there. Defaults to file.
• index.STRUCTURENAME.bsearchshortcut - depicts if the String lookups can be sped up using a binary search shortcut. Possible values are {charmap,default}. Charmap means a hashmap object will be read into memory that defines where to look for a given starting character of the lookup string.

Since:

3.0

Author:

Craig Macdonald

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FSOMapFileLexicon.MapFileLexiconEntryIterator Iterate through the values in order
static class	FSOMapFileLexicon.MapFileLexiconIterator An iterator over the lexicon

Nested classes/interfaces inherited from class org.terrier.structures.MapLexicon

MapLexicon.Id2EntryIndexLookup, MapLexicon.IdIsIndex

Field Summary

Fields inherited from class org.terrier.structures.MapLexicon

keyFactory, map, modificationLock

Constructor Summary

Constructors

Constructor and Description
FSOMapFileLexicon(String structureName, IndexOnDisk index) Construct a new FSOMapFileLexicon
FSOMapFileLexicon(String structureName, String path, String prefix, FixedSizeWriteableFactory<org.apache.hadoop.io.Text> _keyFactory, FixedSizeWriteableFactory<LexiconEntry> _valueFactory, String termIdLookup, String termLookup, String dataFile) Construct an instance of the class with

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
static String	constructFilename(String structureName, String path, String prefix, String extension) Constructs a filename
static void	deleteMapFileLexicon(String structureName, String path, String prefix) Delete a FSOMapFileLexicon within the specified index location
Iterator<Map.Entry<String,LexiconEntry>>	iterator()
static void	optimise(String structureName, IndexOnDisk index, LexiconBuilder.CollectionStatisticsCounter statsCounter) Does two things to a FSOMapFileLexicon: adds the termid lookup file (if required), and also creates the lexicon has file.
static void	optimise(String structureName, IndexOnDisk index, LexiconBuilder.CollectionStatisticsCounter statsCounter, int numEntries) optimise
static void	renameMapFileLexicon(String SrcStructureName, String SrcPath, String SrcPrefix, String destStructureName, String destPath, String destPrefix) Rename a FSOMapFileLexicon within the specified index location

Methods inherited from class org.terrier.structures.MapLexicon

getIthLexiconEntry, getLexiconEntry, getLexiconEntry, getLexiconEntryRange, numberOfEntries, setTermIdLookup

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FSOMapFileLexicon

public FSOMapFileLexicon(String structureName,
                 IndexOnDisk index)
                  throws IOException

Construct a new FSOMapFileLexicon

Throws:

IOException

FSOMapFileLexicon

public FSOMapFileLexicon(String structureName,
                 String path,
                 String prefix,
                 FixedSizeWriteableFactory<org.apache.hadoop.io.Text> _keyFactory,
                 FixedSizeWriteableFactory<LexiconEntry> _valueFactory,
                 String termIdLookup,
                 String termLookup,
                 String dataFile)
                  throws IOException

Construct an instance of the class with

Parameters:

structureName -

path -

prefix -

_keyFactory -

_valueFactory -

termIdLookup -

termLookup -

dataFile -

Throws:

IOException

Method Detail

close

public void close()
           throws IOException

Description copied from class: MapLexicon

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class MapLexicon

Throws:

IOException

iterator

public Iterator<Map.Entry<String,LexiconEntry>> iterator()

optimise

public static void optimise(String structureName,
            IndexOnDisk index,
            LexiconBuilder.CollectionStatisticsCounter statsCounter)
                     throws IOException

Does two things to a FSOMapFileLexicon: adds the termid lookup file (if required), and also creates the lexicon has file.

Parameters:

structureName - - name of the index structure that this FSOMapFileLexicon represents

index - - the index that the index belongs

Throws:

IOException - if an IO problem occurs

optimise

public static void optimise(String structureName,
            IndexOnDisk index,
            LexiconBuilder.CollectionStatisticsCounter statsCounter,
            int numEntries)
                     throws IOException

optimise

Parameters:

structureName -

index -

statsCounter -

numEntries -

Throws:

IOException

constructFilename

public static String constructFilename(String structureName,
                       String path,
                       String prefix,
                       String extension)

Constructs a filename

Parameters:

structureName -

path -

prefix -

extension -

Returns:

filename

renameMapFileLexicon

public static void renameMapFileLexicon(String SrcStructureName,
                        String SrcPath,
                        String SrcPrefix,
                        String destStructureName,
                        String destPath,
                        String destPrefix)

Rename a FSOMapFileLexicon within the specified index location

deleteMapFileLexicon

public static void deleteMapFileLexicon(String structureName,
                        String path,
                        String prefix)

Delete a FSOMapFileLexicon within the specified index location