Terrier Core

The issue here is that while we sort runs within a single reducer by split number, partitions are calculated using map tasks, not by splits. Instead, we should use split information to decide which map output goes to which reduce task.

initial version of a patch for this issue. Requires testing.

new improved patch

tested both docids and meta for ordering on a 7 file subset using 7 maps and 3 reducers, looks ok

tested using wt2g, indexing took 333seconds
retrieval was tested using In_expB2c10.99, Average Precision: 0.3140 - identical to expected value from test index

However, there seem to be a few missing terms
terms=1002586 (MR) vs 1002691 (classical)
this might just be an encoding issue though

Can we get rid of MapEmitted term then if we use SplitEmittedTerm. Perhaps SplitEmittedTerm isnt the best name? Not sure.

Thanks for working on this. I will also give a test.

Yes, MapEmittedTerm is no longer used, similarly MapEmittedTermByMapPartitioner and MapEmittedTermBySplitPartitioner are also unused.

Minor update:

• Use Hadoop writable methods for SplitEmittedTerm (e.g. VInt, as split id and flushes will mostly be small)
• Added test cases for SplitEmittedTerm, particularly with various comparator classes, and the partitioner
• Removed unused class.

Testing for ClueWeb09 showed successful indexing and correct retrieval performance with the correct ordering of docids

Please give a look to see how I designed the test class.

+1 Test Case (that must have taken some time to write!)

As a note, we need to put in the copyright blurb since this is part of the core

+1 also.

Committed to trunk, rev 2746. Thanks Richard!