Terrier Core

Created PositionAwareSplit extends InputSplit
Created PositionAwareDocument extends Document
CollectionRecordReader etc now use these rather than InputSplit and Document
Maps now store the splitnum to the MapDataFile
MapData is now comparable on splitnum
Reducers sort MapData
HadoopRunsMerger now checks against the splitnum rather than MapNo.

Swapped class files for patch

Hi Richard,

This patch looks OK.However, you need to make some consideration for the sorting of DocumentIndexes etc.

I would suggest that closeReduce() stage 3 needs to sort the indices in some way.

Perhaps you wish to write the splt number into the currentIndex object in closeMap(), and then use that to sort the indices array in closeReduce()?

Ta

C

Added sorting on document Index - sorts MapTaskIDs during the load of run files to be inorder of splits using a new IDComparator class

typo fix

Another fix, PositionAwareSplit is now generic on type <T extends InputSplit>
T's class name is written with the file and then used to call the generic constructor for T so that it can do the rest of the reading.

Hi Richard,

Couple of picky things:
1. IDComparator

• Its fairly unusual that a comparator object has state. Can you avoid this?
• Can you add Javadoc so we understand how this class sorts things

2. PositionAwareSplit:

• I'm ok with using the class name to determine the split type
• However, I dont see why it has to be generic on the split type, as getSplit() returns InputSplit
• Can you throw a terrier.utility.WrappedIOException if an except occurs loading the object.
• Use this to load the class without a cast:

  Class<? extends InputSplit> c = Class.forName(className, false, this.getClass().getClassLoader()).asSubclass(InputSplit.class);
  split = c.getConstructor(new Class[0]).newInstance(new Object[0]);

• Why the getConstructor() call? Just use Class.newInstance() instead, as that uses the default constructor.

We need to have some kind of test case to ensure that this works as expected.

Bug found in run matching to run data files during the reduce step.

Improved patch. Appears to fix docids properly. Prevents double flushes, and keeps all indexer variables at correct values.

Richard,

I'm wondering if this patch could be back-ported to Terrier 2, as block indexing is taking an age. (Dont do anything yet)

C

Last revision of patch. This patch for trunk only.

Patch committed for trunk.

patch for 2.x SVN branch (rev 2573). All tests pass.