Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-212

Evaluation doesn't support graded relevance judgements

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 3.5
    • Fix Version/s: 3.6
    • Component/s: .evaluation
    • Labels:
      None

      Description


      TRECQrelsInMemory.loadQrelsFile() method accepts wrong documents in
      qrels file as relevant such that

      "boolean relevant = !(relGrade == 0);".

      It should be "boolean relevant = (relGrade > 0);", because in the
      current Web track qrels, "junk" documents are marked as minus 2 (-2),
      which is also different from 0.

      [identified by B. Taner DINCER]

        Attachments

          Activity

          craigm Craig Macdonald created issue -
          Hide
          craigm Craig Macdonald added a comment -

          Committed, with unit test, r3666. Thanks taner.

          Show
          craigm Craig Macdonald added a comment - Committed, with unit test, r3666. Thanks taner.
          craigm Craig Macdonald made changes -
          Field Original Value New Value
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              craigm Craig Macdonald
              Reporter:
              craigm Craig Macdonald
            • Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: