Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-38

MapReduce InputFormat for BitPostingIndexInputStream

          Details

          • Type: Improvement
          • Status: Resolved
          • Priority: Major
          • Resolution: Fixed
          • Affects Version/s: 3.0
          • Fix Version/s: 3.0
          • Component/s: .structures
          • Labels:
            None

            Description

            Recent core changes have made generic PostingIndex and PostingIndexInputStream objects which give access to IterablePostings. It would be good to have an InputFormat for splitting the reading of a PostingIndex across various map tasks.

              Attachments

                Activity

                Ascending order - Click to sort in descending order
                Hide
                Permalink
                craigm Craig Macdonald added a comment -

                This would have benefits in the following scenarios:

                • If we use PostingIndex as a LinkServer, then this would allow link analysis index processing to be easily split
                • Inversion of indices (i.e. Inverted-> Direct; Direct->Inverted) could be split to run in parallel
                • DirectIndex analysis.

                As a PostingIndex can be large, locality should be supported.

                Show
                craigm Craig Macdonald added a comment - This would have benefits in the following scenarios: If we use PostingIndex as a LinkServer, then this would allow link analysis index processing to be easily split Inversion of indices (i.e. Inverted-> Direct; Direct->Inverted) could be split to run in parallel DirectIndex analysis. As a PostingIndex can be large, locality should be supported.
                Hide
                Permalink
                craigm Craig Macdonald added a comment -

                Initial version, untested.

                Show
                craigm Craig Macdonald added a comment - Initial version, untested.
                Hide
                Permalink
                craigm Craig Macdonald added a comment -

                Updated version - some files were missing from the patch.

                Show
                craigm Craig Macdonald added a comment - Updated version - some files were missing from the patch.
                Hide
                Permalink
                craigm Craig Macdonald added a comment -

                Resolved a final version to trunk.

                Show
                craigm Craig Macdonald added a comment - Resolved a final version to trunk.

                  People

                  • Assignee:
                    craigm Craig Macdonald
                    Reporter:
                    craigm Craig Macdonald
                  • Watchers:
                    2 Start watching this issue

                    Dates

                    • Created:
                      Updated:
                      Resolved: