[TR-212] Evaluation doesn't support graded relevance judgements Created: 03/Aug/12  Updated: 03/Aug/12  Resolved: 03/Aug/12

Status: Resolved
Project: Terrier Core
Component/s: .evaluation
Affects Version/s: 3.5
Fix Version/s: 3.6

Type: Bug Priority: Trivial
Reporter: Craig Macdonald Assignee: Craig Macdonald
Resolution: Fixed  
Labels: None


TRECQrelsInMemory.loadQrelsFile() method accepts wrong documents in
qrels file as relevant such that

"boolean relevant = !(relGrade == 0);".

It should be "boolean relevant = (relGrade > 0);", because in the
current Web track qrels, "junk" documents are marked as minus 2 (-2),
which is also different from 0.

[identified by B. Taner DINCER]

Comment by Craig Macdonald [ 03/Aug/12 ]

Committed, with unit test, r3666. Thanks taner.

Generated at Sat Jan 23 08:54:13 GMT 2021 using JIRA 7.1.1#71004-sha1:d6b2c0d9b7051e9fb5e4eb8ce177ca56d91d7bd8.