public class HadoopRunsMerger extends RunsMerger
RunsMerger.PostingComparator
Modifier and Type | Field and Description |
---|---|
protected LinkedList<MapData> |
mapData
The data loaded from side-effect files about each map task
|
protected int |
numReducers
Number of Reducers Used
|
bos, currentTerm, lastDocFreq, lastDocument, lastFreq, lastTermWritten, myRun, numberOfPointers, queue, runsSource, startOffset, termStatistics
Constructor and Description |
---|
HadoopRunsMerger(RunIteratorFactory _runsSource)
Constructs an instance of HadoopRunsMerger.
|
Modifier and Type | Method and Description |
---|---|
void |
beginMerge(LinkedList<MapData> _mapData)
Alternate Merge operation for merging a linked list of runs of the form
Hadoop_MapData.
|
void |
endMerge(LexiconOutputStream<String> lexStream)
Ends the merging phase, writes the last entry and closes the streams.
|
int |
getDocumentOffset(int splitNo,
int flushNumber)
Get the offset for the document based on a split and flush.
|
int |
getNumReducers()
Gets the number of Reducers to Merge for:
1 for single Reducer,
>1 for multi-Reducers
|
void |
mergeOne(LexiconOutputStream<String> lexStream)
Mergers one term in the runs.
|
void |
setNumReducers(int _numReducers)
Sets the number of Reducers to Merge for:
1 for single Reducer,
>1 for multi-Reducers
|
beginMerge, getBitOffset, getBos, getByteOffset, getLastDocFreq, getLastFreq, getLastTermWritten, getNumberOfPointers, getNumberOfTerms, init, init, isDone, setBos, setLastTermWritten
protected LinkedList<MapData> mapData
protected int numReducers
public HadoopRunsMerger(RunIteratorFactory _runsSource)
_runsSource
- public void beginMerge(LinkedList<MapData> _mapData)
_mapData
- - information about the number of documents per map and run. One element for every map.IOException
public void endMerge(LexiconOutputStream<String> lexStream)
endMerge
in class RunsMerger
lexStream
- LexiconOutputStream used to write the lexicon.public void mergeOne(LexiconOutputStream<String> lexStream) throws Exception
mergeOne
in class RunsMerger
lexStream
- LexiconOutputStream used to write the lexicon.Exception
- if an I/O error occurs.public int getNumReducers()
public void setNumReducers(int _numReducers)
public int getDocumentOffset(int splitNo, int flushNumber) throws IOException
IOException
Terrier 4.0. Copyright © 2004-2014 University of Glasgow