Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-212

Evaluation doesn't support graded relevance judgements

          Details

          • Type: Bug
          • Status: Resolved
          • Priority: Trivial
          • Resolution: Fixed
          • Affects Version/s: 3.5
          • Fix Version/s: 3.6
          • Component/s: .evaluation
          • Labels:
            None

            Description


            TRECQrelsInMemory.loadQrelsFile() method accepts wrong documents in
            qrels file as relevant such that

            "boolean relevant = !(relGrade == 0);".

            It should be "boolean relevant = (relGrade > 0);", because in the
            current Web track qrels, "junk" documents are marked as minus 2 (-2),
            which is also different from 0.

            [identified by B. Taner DINCER]

              Attachments

                Activity

                Ascending order - Click to sort in descending order
                craigm Craig Macdonald created issue -
                Hide
                Permalink
                craigm Craig Macdonald added a comment -

                Committed, with unit test, r3666. Thanks taner.

                Show
                craigm Craig Macdonald added a comment - Committed, with unit test, r3666. Thanks taner.
                craigm Craig Macdonald made changes -
                Field Original Value New Value
                Status Open [ 1 ] Resolved [ 5 ]
                Resolution Fixed [ 1 ]

                  People

                  • Assignee:
                    craigm Craig Macdonald
                    Reporter:
                    craigm Craig Macdonald
                  • Watchers:
                    1 Start watching this issue

                    Dates

                    • Created:
                      Updated:
                      Resolved: