Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-119

Temporary jar/properties in HDFS /tmp are not deleted

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2, 2.2.1, 3.0
    • Fix Version/s: 3.5
    • Component/s: .utility
    • Labels:
      None

      Description

      We rely on Hadoop's FileSystem.deleteOnExit() method to delete temporary jar/property files that are uploaded to HDFS /tmp folder. However, this doesn't seem to ever work.

      We should instead add a HadoopUtility.finishTerrierJob(JobConf) to do the opposite of HadoopUtility.finishTerrierJob();

        Attachments

          Activity

          Hide
          craigm Craig Macdonald added a comment -

          This has been committed to trunk.

          Show
          craigm Craig Macdonald added a comment - This has been committed to trunk.
          Hide
          craigm Craig Macdonald added a comment -

          Final patch for this issue. HadoopUtility.finishTerrierJob(JobConf) should be called after each MR job has finished.

          Show
          craigm Craig Macdonald added a comment - Final patch for this issue. HadoopUtility.finishTerrierJob(JobConf) should be called after each MR job has finished.

            People

            • Assignee:
              craigm Craig Macdonald
              Reporter:
              craigm Craig Macdonald
            • Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: