org.terrier.realtime.multi

Class MultiLexicon

java.lang.Object

org.terrier.structures.Lexicon<String>

org.terrier.realtime.multi.MultiLexicon

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<Map.Entry<String,LexiconEntry>>

public class MultiLexicon
extends Lexicon<String>

A Lexicon index structure for use with a MultiIndex. It wraps around multiple lexicons from different index shards. IMPORTANT: Not all lexicon access methods are supported since a lexicon entry can appear in any number of lexicons! This has the following consequences:

• A MultiLexicon can not be iterated over without doing a temporary merge of all lexicon structures if we are to have up-to-date statistics, this is supported but may be slow
• getIthLexiconEntry() is not supported (the contents of the ith entry can change over time as new documents are added)
• The unique number of terms is not stored and needs to be calculated on-the-fly.

Properties

• MultiLexicon.approxNumEntries - do we try and approximate the number of lexicon entries (saves a lot of time but is inaccurate), default is true.
• MultiLexicon.updateTermListOnIteratorCreate - do we re-build the full list of terms in the lexicon when an iterator is created. This can be slow, but we might miss new terms otherwise.

Since:

4.0

Author:

Richard McCreadie, Stuart Mackie

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	MultiLexicon.LexiconEntryIterator

Nested classes/interfaces inherited from class org.terrier.structures.Lexicon

Lexicon.LexiconFileEntry<KEY2>

Constructor Summary

Constructors

Constructor and Description
MultiLexicon(Lexicon<String>[] lexicons, int[] numTerms) constructor.

Method Summary

Modifier and Type	Method and Description
void	close() Close all of the contained lexicons.
Lexicon<String>	getIthLexicon(int index)
Map.Entry<String,LexiconEntry>	getIthLexiconEntry(int index) This is an invalid method since a lexicon entry can appear in any number of lexicons.
Map.Entry<String,LexiconEntry>	getLexiconEntry(int termid) Get the term and LexiconEntry (contains statistics and a pointer) for a given term id.
LexiconEntry	getLexiconEntry(String term) Get the LexiconEntry (contains statistics and a pointer) for the given term.
Iterator<Map.Entry<String,LexiconEntry>>	getLexiconEntryRange(String from, String to) Returns an iterator over a set of LexiconEntries within a range of entries in the lexicon.
static int	hashCode(String term)
static char	hashCodePrefix(int hashcode)
Iterator<Map.Entry<String,LexiconEntry>>	iterator() Creates an iterator over the MultiLexicon structure.
int	numberOfEntries() Return the number of entries in the lexicon

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

MultiLexicon

public MultiLexicon(Lexicon<String>[] lexicons,
                    int[] numTerms)

constructor.

Method Detail

getIthLexicon

public Lexicon<String> getIthLexicon(int index)

hashCode

public static int hashCode(String term)

hashCodePrefix

public static char hashCodePrefix(int hashcode)

numberOfEntries

public int numberOfEntries()

Return the number of entries in the lexicon

Specified by:

numberOfEntries in class Lexicon<String>

getLexiconEntry

public LexiconEntry getLexiconEntry(String term)

Get the LexiconEntry (contains statistics and a pointer) for the given term. Returns null if the term is not found in the lexicon.

Specified by:

getLexiconEntry in class Lexicon<String>

Parameters:

term - key to lookup the lexicon with

Returns:

LexiconEntry for that term, or null if not found

getLexiconEntry

public Map.Entry<String,LexiconEntry> getLexiconEntry(int termid)

Get the term and LexiconEntry (contains statistics and a pointer) for a given term id. Throws NoSuchElementException is the termid is not found.

Specified by:

getLexiconEntry in class Lexicon<String>

Parameters:

termid - the term id to lookup in the lexicon

Returns:

Map.Entry tuple containing the term and the LexiconEntry

getIthLexiconEntry

public Map.Entry<String,LexiconEntry> getIthLexiconEntry(int index)

This is an invalid method since a lexicon entry can appear in any number of lexicons. In general DO NOT USE THIS! This method is only implemented such that a random term can be chosen within the JUnit tests.

Specified by:

getIthLexiconEntry in class Lexicon<String>

Parameters:

index - the entry number to lookup in the lexicon

Returns:

Map.Entry tuple containing the term and the LexiconEntry

iterator

public Iterator<Map.Entry<String,LexiconEntry>> iterator()

Creates an iterator over the MultiLexicon structure. Iteration is in alphabetical order, if MultiLexicon.approxNumEntries is set to false then the first time this is called will result in a full scan of each lexicon.

close

public void close()
           throws IOException

Close all of the contained lexicons.

Throws:

IOException

getLexiconEntryRange

public Iterator<Map.Entry<String,LexiconEntry>> getLexiconEntryRange(String from,
                                                                     String to)

Description copied from class: Lexicon

Returns an iterator over a set of LexiconEntries within a range of entries in the lexicon.

Specified by:

getLexiconEntryRange in class Lexicon<String>

Parameters:

from - low endpoint term in the subset, inclusive.

to - high endpoint term in the subset, exclusive.

Returns:

Iterator over the set.