org.terrier.structures.collections
Class FSOrderedMapFile<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

java.lang.Object
  org.terrier.structures.collections.FSOrderedMapFile<K,V>

Type Parameters:

K - Type of the keys

V - Type of the values

All Implemented Interfaces:

java.io.Closeable, java.util.Map<K,V>, OrderedMap<K,V>

public class FSOrderedMapFile<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>
extends java.lang.Object
implements OrderedMap<K,V>, java.io.Closeable

An implementation of java.util.Map that can be accessed from disk. Key and value types are assumed to have a fixed size. Their factories must be passed to the constructor. In the name, FSOrderedMapFile, FS stands for Fixed Size.

Since:

3.0

Author:

Craig Macdonald

Nested Class Summary

static class	FSOrderedMapFile.EntryIterator<IK extends org.apache.hadoop.io.Writable,IV extends org.apache.hadoop.io.Writable> an iterator for entries.
static interface	FSOrderedMapFile.FSOMapFileBSearchShortcut<KEY> interface FSOMapFileBSearchShortcut
static class	FSOrderedMapFile.MapFileInMemory<IK extends org.apache.hadoop.io.Writable,IV extends org.apache.hadoop.io.Writable> MapFileInMemory class
static interface	FSOrderedMapFile.MapFileWriter Interface for writing a FSOMapFile
static class	FSOrderedMapFile.MultiFSOMapWriter Writes out a FSOMapFile, but assumes that input data need not be sorted by key.

Nested classes/interfaces inherited from interface java.util.Map

java.util.Map.Entry<K,V>

Field Summary

protected RandomDataInput	dataFile actual underlying data file
protected java.lang.String	dataFilename filename of the underlying file
protected int	entrySize total size of one key,value pair
protected FixedSizeWriteableFactory<K>	keyFactory
protected static org.apache.log4j.Logger	logger The logger used for this class
protected int	numberOfEntries The number of entries in the file.
protected FSOrderedMapFile.FSOMapFileBSearchShortcut<K>	shortcut
static java.lang.String	USUAL_EXTENSION USUAL_EXTENSION
protected FixedSizeWriteableFactory<V>	valueFactory

Constructor Summary

FSOrderedMapFile(Index index, java.lang.String structureName)
constructor

FSOrderedMapFile(RandomDataInput file, java.lang.String filename, FixedSizeWriteableFactory<K> _keyFactory, FixedSizeWriteableFactory<V> _valueFactory)
constructor

FSOrderedMapFile(java.lang.String filename, boolean updateable, FixedSizeWriteableFactory<K> _keyFactory, FixedSizeWriteableFactory<V> _valueFactory)
Construct a new object to access the underlying file data structure

Method Summary

protected void	_clear()
void	clear() Remove all entries from this map
void	close()
boolean	containsKey(java.lang.Object o)
boolean	containsValue(java.lang.Object o)
java.util.Set<java.util.Map.Entry<K,V>>	entrySet()
java.util.Map.Entry<K,V>	get(int entryNumber) Return the entry at the specified index
V	get(java.lang.Object _key)
protected java.util.Map.Entry<K,V>	getEntry(K key) this method is the one which does the actual disk lookup of entries
WriteableFactory<K>	getKeyFactory() Get the key factory
WriteableFactory<V>	getValueFactory() Get the value factory
boolean	isEmpty()
java.util.Set<K>	keySet()
static FSOrderedMapFile.MapFileWriter	mapFileWrite(java.lang.String filename) returns a utility class which can be used to write a FSOrderedMapFile.
static void	mapFileWrite(java.lang.String filename, java.lang.Iterable<java.util.Map.Entry<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable>> t) writes an entire map FSOrderedMapFile at once, to the specified filename, and using the data contained in the specified iterator
static void	mapFileWrite(java.lang.String filename, java.util.Iterator<java.util.Map.Entry<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable>> ti) writes an entire map FSOrderedMapFile at once, to the specified filename, and using the data contained in the specified iterator
static int	numberOfEntries(java.lang.String filename, FixedSizeWriteableFactory<?> _keyFactory, FixedSizeWriteableFactory<?> _valueFactory) Return number of entries
V	put(K key, V value)
void	putAll(java.util.Map<? extends K,? extends V> m)
V	remove(java.lang.Object _key)
void	setBSearchShortcut(FSOrderedMapFile.FSOMapFileBSearchShortcut<K> _shortcut) Set the FSOMapFileBSearchShortcut
int	size() Returns the number of entries in this map
java.util.Collection<V>	values()
protected RandomDataOutput	write()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Map

equals, hashCode

Field Detail

USUAL_EXTENSION

public static final java.lang.String USUAL_EXTENSION

USUAL_EXTENSION

See Also:

Constant Field Values

logger

protected static final org.apache.log4j.Logger logger

The logger used for this class

dataFile

protected RandomDataInput dataFile

actual underlying data file

dataFilename

protected java.lang.String dataFilename

filename of the underlying file

numberOfEntries

protected int numberOfEntries

The number of entries in the file.

entrySize

protected int entrySize

total size of one key,value pair

shortcut

protected FSOrderedMapFile.FSOMapFileBSearchShortcut<K extends org.apache.hadoop.io.WritableComparable> shortcut

keyFactory

protected FixedSizeWriteableFactory<K extends org.apache.hadoop.io.WritableComparable> keyFactory

valueFactory

protected FixedSizeWriteableFactory<V extends org.apache.hadoop.io.Writable> valueFactory

Constructor Detail

FSOrderedMapFile

public FSOrderedMapFile(Index index,
                        java.lang.String structureName)
                 throws java.io.IOException

constructor

Parameters:

index -

structureName -

Throws:

java.io.IOException

FSOrderedMapFile

public FSOrderedMapFile(java.lang.String filename,
                        boolean updateable,
                        FixedSizeWriteableFactory<K> _keyFactory,
                        FixedSizeWriteableFactory<V> _valueFactory)
                 throws java.io.IOException

Construct a new object to access the underlying file data structure

Parameters:

filename - Filename of the file containing the structure

updateable - Whether the file can be updated in this JVM

_keyFactory - factory object for keys

_valueFactory - factory object for values

Throws:

java.io.IOException - thrown if an IO problem occurs

FSOrderedMapFile

public FSOrderedMapFile(RandomDataInput file,
                        java.lang.String filename,
                        FixedSizeWriteableFactory<K> _keyFactory,
                        FixedSizeWriteableFactory<V> _valueFactory)
                 throws java.io.IOException

constructor

Parameters:

file -

filename -

_keyFactory -

_valueFactory -

Throws:

java.io.IOException

Method Detail

write

protected RandomDataOutput write()

numberOfEntries

public static int numberOfEntries(java.lang.String filename,
                                  FixedSizeWriteableFactory<?> _keyFactory,
                                  FixedSizeWriteableFactory<?> _valueFactory)

Return number of entries

Parameters:

filename -

_keyFactory -

_valueFactory -

Returns:

number of entries

getKeyFactory

public WriteableFactory<K> getKeyFactory()

Get the key factory

getValueFactory

public WriteableFactory<V> getValueFactory()

Get the value factory

clear

public void clear()

Remove all entries from this map

Specified by:

clear in interface java.util.Map<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

_clear

protected void _clear()

entrySet

public java.util.Set<java.util.Map.Entry<K,V>> entrySet()

Specified by:

entrySet in interface java.util.Map<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

keySet

public java.util.Set<K> keySet()

Specified by:

keySet in interface java.util.Map<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

values

public java.util.Collection<V> values()

Specified by:

values in interface java.util.Map<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

size

public int size()

Returns the number of entries in this map

Specified by:

size in interface java.util.Map<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

containsValue

public boolean containsValue(java.lang.Object o)

Specified by:

containsValue in interface java.util.Map<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

containsKey

public boolean containsKey(java.lang.Object o)

Specified by:

containsKey in interface java.util.Map<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

isEmpty

public boolean isEmpty()

Specified by:

isEmpty in interface java.util.Map<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

put

public V put(K key,
             V value)

Specified by:

put in interface java.util.Map<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

remove

public V remove(java.lang.Object _key)

Specified by:

remove in interface java.util.Map<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

setBSearchShortcut

public void setBSearchShortcut(FSOrderedMapFile.FSOMapFileBSearchShortcut<K> _shortcut)

Set the FSOMapFileBSearchShortcut

getEntry

protected java.util.Map.Entry<K,V> getEntry(K key)

this method is the one which does the actual disk lookup of entries

get

public V get(java.lang.Object _key)

Specified by:

get in interface java.util.Map<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

get

public java.util.Map.Entry<K,V> get(int entryNumber)

Return the entry at the specified index

Specified by:

get in interface OrderedMap<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

putAll

public void putAll(java.util.Map<? extends K,? extends V> m)

Specified by:

putAll in interface java.util.Map<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Throws:

java.io.IOException

mapFileWrite

public static void mapFileWrite(java.lang.String filename,
                                java.lang.Iterable<java.util.Map.Entry<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable>> t)
                         throws java.io.IOException

writes an entire map FSOrderedMapFile at once, to the specified filename, and using the data contained in the specified iterator

Throws:

java.io.IOException

mapFileWrite

public static void mapFileWrite(java.lang.String filename,
                                java.util.Iterator<java.util.Map.Entry<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable>> ti)
                         throws java.io.IOException

writes an entire map FSOrderedMapFile at once, to the specified filename, and using the data contained in the specified iterator

Throws:

java.io.IOException

mapFileWrite

public static FSOrderedMapFile.MapFileWriter mapFileWrite(java.lang.String filename)
                                                   throws java.io.IOException

returns a utility class which can be used to write a FSOrderedMapFile. Input data MUST be sorted by key.

Throws:

java.io.IOException