Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-59

Reset problem in Terrier evaluation tool

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0
    • Fix Version/s: 3.0
    • Component/s: None
    • Labels:
      None

      Description

      Nicola reported today a bug with the Terrier evaluation tool. For some queries, the system reports an MAP of 2 (yes 2) at precision @1. It appears that it is a resetting problem when multiple evaluation files are used.

        Attachments

          Activity

          Hide
          craigm Craig Macdonald added a comment -

          Not a reset problem actually. A problem calculating precision at rank.

          if (relevantRetrieved[j].rank <= precisionRank)
          

          should read

          if (relevantRetrieved[j].rank < precisionRank)
          

          because ranks are 0 based.

          I have also added test cases for the AdhocEvaluation class. The test cases check various measures (hand calculated) for rankings upto size 4. Without the patch, the test fails. With it, the test passes.

          Show
          craigm Craig Macdonald added a comment - Not a reset problem actually. A problem calculating precision at rank. if (relevantRetrieved[j].rank <= precisionRank) should read if (relevantRetrieved[j].rank < precisionRank) because ranks are 0 based. I have also added test cases for the AdhocEvaluation class. The test cases check various measures (hand calculated) for rankings upto size 4. Without the patch, the test fails. With it, the test passes.
          Hide
          ounis Iadh Ounis added a comment -

          Well spotted Craig!

          Show
          ounis Iadh Ounis added a comment - Well spotted Craig!

            People

            • Assignee:
              craigm Craig Macdonald
              Reporter:
              ounis Iadh Ounis
            • Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: