Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-344

Inverted2DirectIndexBuilder fails for large corpora where a partition does not contain any postings

          Details

          • Type: Bug
          • Status: Reopened
          • Priority: Major
          • Resolution: Unresolved
          • Affects Version/s: 4.0
          • Fix Version/s: 4.1
          • Component/s: .structures
          • Labels:
            None

            Description

            We found this for .gov2 corpus:

            INFO - Generating postings for documents with ids 0 to 166153
            ERROR - Couldnt create a direct structure from the inverted structure
            java.lang.ArrayIndexOutOfBoundsException: 990245
            at org.terrier.structures.indexing.singlepass.Inverted2DirectIndexBuilder.traverseInvertedFile(Inverted2DirectIndexBuilder.java:340)
            at org.terrier.structures.indexing.singlepass.Inverted2DirectIndexBuilder.createDirectIndex(Inverted2DirectIndexBuilder.java:168)
            at org.terrier.structures.indexing.singlepass.Inverted2DirectIndexBuilder.main(Inverted2DirectIndexBuilder.java:408)
            at org.terrier.applications.TrecTerrier.run(TrecTerrier.java:534)
            at org.terrier.applications.TrecTerrier.applyOptions(TrecTerrier.java:588)
            at org.terrier.applications.TrecTerrier.main(TrecTerrier.java:245)

              Attachments

                Issue Links

                is duplicated by

                Bug - A problem which impairs or prevents the functions of the product. TR-338 Issue in class Inverted2DirectIndexBuilder

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved

                  Activity

                  Ascending order - Click to sort in descending order
                  Hide
                  Permalink
                  craigm Craig Macdonald added a comment -

                  Patch, including test cases (and other necessary changes), along with some other readability improvements to the Inverted2Direct.

                  Show
                  craigm Craig Macdonald added a comment - Patch, including test cases (and other necessary changes), along with some other readability improvements to the Inverted2Direct.
                  Hide
                  Permalink
                  craigm Craig Macdonald added a comment -

                  Committed to git

                  Show
                  craigm Craig Macdonald added a comment - Committed to git
                  Hide
                  Permalink
                  richardm Richard McCreadie added a comment -

                  TestInverted2DirectIndexBuilder appears to have been commited to the root directory, not to src folder.

                  Re-opening issue until fixed.

                  Show
                  richardm Richard McCreadie added a comment - TestInverted2DirectIndexBuilder appears to have been commited to the root directory, not to src folder. Re-opening issue until fixed.

                    People

                    • Assignee:
                      craigm Craig Macdonald
                      Reporter:
                      craigm Craig Macdonald
                    • Watchers:
                      0 Start watching this issue

                      Dates

                      • Created:
                        Updated: