Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-44

In singlepass indexing, checking for enough free memory is insufficient

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0
    • Fix Version/s: 3.0
    • Component/s: .indexing
    • Labels:
      None

      Description

      Single-pass tries to use as much memory as possible for mini-inverted indices (flushes). It uses some Java code to guess how much memory is left.
      When JVM has allocated all memory, and when only 70% free, flush().

      However, Java's memory management isn't reliable. We can easily get out of memory errors, particularly in Hadoop mode, and often for block indexing. Java6 makes this problem worse as it will throw OutOfMemoryError earlier than Java 5 would.

        Attachments

          Activity

          craigm Craig Macdonald created issue -
          craigm Craig Macdonald made changes -
          Field Original Value New Value
          Attachment singlepass-used-memory.patch [ 10144 ]
          craigm Craig Macdonald made changes -
          Assignee Iadh Ounis [ ounis ] Craig Macdonald [ craigm ]
          craigm Craig Macdonald made changes -
          Component/s Core [ 10020 ]
          craigm Craig Macdonald made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          craigm Craig Macdonald made changes -
          Affects Version/s 3.0 [ 10020 ]
          Fix Version/s 3.0 [ 10020 ]
          craigm Craig Macdonald made changes -
          Project TREC [ 10010 ] Terrier Core [ 10000 ]
          Key TREC-43 TR-44
          Workflow jira [ 10112 ] Terrier Open Source [ 10302 ]
          Affects Version/s 3.0 [ 10030 ]
          Affects Version/s 3.0 [ 10020 ]
          Component/s .indexing [ 10002 ]
          Component/s Core [ 10020 ]
          Fix Version/s 3.0 [ 10030 ]
          Fix Version/s 3.0 [ 10020 ]

            People

            • Assignee:
              craigm Craig Macdonald
              Reporter:
              craigm Craig Macdonald
            • Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: