Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-41

Hadoop Indexing loads CompressedMetaIndex into memory during reduce phase

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0
    • Fix Version/s: 3.0
    • Component/s: None
    • Labels:
      None

      Description

      The use of the MetaIndex during the reduce phase to merge the meta indexes. If there are lots of map tasks, this can result in too many meta indices being loaded into memory at once.

      The solution is to access the MetaIndex as a stream.

        Attachments

          Activity

          craigm Craig Macdonald created issue -
          craigm Craig Macdonald made changes -
          Field Original Value New Value
          Attachment TR-30.v1.patch [ 10113 ]
          craigm Craig Macdonald made changes -
          Project Terrier Core [ 10000 ] TREC [ 10010 ]
          Key TR-30 TREC-36
          Issue Type Bug [ 1 ] Improvement [ 4 ]
          Workflow Terrier Open Source [ 10102 ] jira [ 10105 ]
          craigm Craig Macdonald made changes -
          Component/s Core [ 10020 ]
          Hide
          craigm Craig Macdonald added a comment -

          Committed to SVN. Also uses MapReduce job to do metaindex inversion, i.e. meta->docid

          Show
          craigm Craig Macdonald added a comment - Committed to SVN. Also uses MapReduce job to do metaindex inversion, i.e. meta->docid
          craigm Craig Macdonald made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          craigm Craig Macdonald made changes -
          Affects Version/s 3.0 [ 10020 ]
          Fix Version/s 3.0 [ 10020 ]
          craigm Craig Macdonald made changes -
          Project TREC [ 10010 ] Terrier Core [ 10000 ]
          Key TREC-36 TR-41
          Workflow jira [ 10105 ] Terrier Open Source [ 10299 ]
          Affects Version/s 3.0 [ 10030 ]
          Affects Version/s 3.0 [ 10020 ]
          Component/s Core [ 10020 ]
          Fix Version/s 3.0 [ 10030 ]
          Fix Version/s 3.0 [ 10020 ]

            People

            • Assignee:
              craigm Craig Macdonald
              Reporter:
              craigm Craig Macdonald
            • Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: