[TR-212] Evaluation doesn't support graded relevance judgements Created: 03/Aug/12 Updated: 03/Aug/12 Resolved: 03/Aug/12
|Reporter:||Craig Macdonald||Assignee:||Craig Macdonald|
TRECQrelsInMemory.loadQrelsFile() method accepts wrong documents in
qrels file as relevant such that
"boolean relevant = !(relGrade == 0);".
It should be "boolean relevant = (relGrade > 0);", because in the
current Web track qrels, "junk" documents are marked as minus 2 (-2),
which is also different from 0.
[identified by B. Taner DINCER]
|Comment by Craig Macdonald [ 03/Aug/12 ]|
Committed, with unit test, r3666. Thanks taner.