Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-344

Inverted2DirectIndexBuilder fails for large corpora where a partition does not contain any postings

    Details

    • Type: Bug
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.0
    • Fix Version/s: 4.1
    • Component/s: .structures
    • Labels:
      None

      Description

      We found this for .gov2 corpus:

      INFO - Generating postings for documents with ids 0 to 166153
      ERROR - Couldnt create a direct structure from the inverted structure
      java.lang.ArrayIndexOutOfBoundsException: 990245
      at org.terrier.structures.indexing.singlepass.Inverted2DirectIndexBuilder.traverseInvertedFile(Inverted2DirectIndexBuilder.java:340)
      at org.terrier.structures.indexing.singlepass.Inverted2DirectIndexBuilder.createDirectIndex(Inverted2DirectIndexBuilder.java:168)
      at org.terrier.structures.indexing.singlepass.Inverted2DirectIndexBuilder.main(Inverted2DirectIndexBuilder.java:408)
      at org.terrier.applications.TrecTerrier.run(TrecTerrier.java:534)
      at org.terrier.applications.TrecTerrier.applyOptions(TrecTerrier.java:588)
      at org.terrier.applications.TrecTerrier.main(TrecTerrier.java:245)

        Attachments

          Issue Links

            Activity

            Hide
            craigm Craig Macdonald added a comment -

            Patch, including test cases (and other necessary changes), along with some other readability improvements to the Inverted2Direct.

            Show
            craigm Craig Macdonald added a comment - Patch, including test cases (and other necessary changes), along with some other readability improvements to the Inverted2Direct.
            Hide
            craigm Craig Macdonald added a comment -

            Committed to git

            Show
            craigm Craig Macdonald added a comment - Committed to git
            Hide
            richardm Richard McCreadie added a comment -

            TestInverted2DirectIndexBuilder appears to have been commited to the root directory, not to src folder.

            Re-opening issue until fixed.

            Show
            richardm Richard McCreadie added a comment - TestInverted2DirectIndexBuilder appears to have been commited to the root directory, not to src folder. Re-opening issue until fixed.

              People

              • Assignee:
                craigm Craig Macdonald
                Reporter:
                craigm Craig Macdonald
              • Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated: