Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-232

HadoopImageTerrier project can't work when using HadoopIndexer.java on a cloudera environment

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Hi,
      I am trying to use ImageTerrier on Hadoop . when i try to run HadoopIndexer.java int package org.imageterrier.indexers.hadoop , i meet many errors and i nearly make no change . I run Terrier-3.5 on a pre-build Cloudera VM,the hadoop version is hadoop-0.20.2-cdh3u6 . The error message shows as follows:
      -----------------------------------------------------------------
      INFO - Total input paths to process : 1
      INFO - Running job: job_201308280427_0013
      INFO - map 0% reduce 0%
      INFO - Task Id : attempt_201308280427_0013_m_000000_0, Status : FAILED
      java.lang.NullPointerException
              at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:94)
              at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
              at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
              at org.apache.hadoop.mapred.Child.main(Child.java:260)

      attempt_201308280427_0013_m_000000_0: WARNING: The file terrier.properties was not found at location /etc/terrier.properties
      attempt_201308280427_0013_m_000000_0: Assuming the value of terrier.home from the corresponding system property.
      attempt_201308280427_0013_m_000000_0: Loading codebook...
      attempt_201308280427_0013_m_000000_0: codebookURL:/usr/local/hadoop_data/mapred_local/taskTracker/hadoop/jobcache/job_201308280427_0013/attempt_201308280427_0013_m_000000_0/work/tmp/imageterrier2475550265630386496.cluster
      attempt_201308280427_0013_m_000000_0: Please ensure that the property terrier.home
      attempt_201308280427_0013_m_000000_0: is specified in the file terrier.properties,
      attempt_201308280427_0013_m_000000_0: or as a system property in the command line.
      attempt_201308280427_0013_m_000000_0: Exception in thread "Thread for syncLogs" java.lang.NullPointerException
      attempt_201308280427_0013_m_000000_0: at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:94)
      attempt_201308280427_0013_m_000000_0: at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
      attempt_201308280427_0013_m_000000_0: at org.apache.hadoop.mapred.Child$3.run(Child.java:153)
      attempt_201308280427_0013_m_000000_0: WARN - Error running child
      attempt_201308280427_0013_m_000000_0: java.lang.NullPointerException
      attempt_201308280427_0013_m_000000_0: at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:94)
      attempt_201308280427_0013_m_000000_0: at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
      attempt_201308280427_0013_m_000000_0: at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
      attempt_201308280427_0013_m_000000_0: at java.security.AccessController.doPrivileged(Native Method)
      attempt_201308280427_0013_m_000000_0: at javax.security.auth.Subject.doAs(Subject.java:396)
      attempt_201308280427_0013_m_000000_0: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
      attempt_201308280427_0013_m_000000_0: at org.apache.hadoop.mapred.Child.main(Child.java:260)
      attempt_201308280427_0013_m_000000_0: INFO - Runnning cleanup for the task
      INFO - Task Id : attempt_201308280427_0013_m_000000_1, Status : FAILED
      java.lang.NullPointerException
              at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:94)
              at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
              at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
              at org.apache.hadoop.mapred.Child.main(Child.java:260)

      attempt_201308280427_0013_m_000000_1: WARNING: The file terrier.properties was not found at location /etc/terrier.properties
      attempt_201308280427_0013_m_000000_1: Assuming the value of terrier.home from the corresponding system property.
      attempt_201308280427_0013_m_000000_1: Loading codebook...
      attempt_201308280427_0013_m_000000_1: codebookURL:/usr/local/hadoop_data/mapred_local/taskTracker/hadoop/jobcache/job_201308280427_0013/attempt_201308280427_0013_m_000000_1/work/tmp/imageterrier4337986671433788438.cluster
      attempt_201308280427_0013_m_000000_1: Please ensure that the property terrier.home
      attempt_201308280427_0013_m_000000_1: is specified in the file terrier.properties,
      attempt_201308280427_0013_m_000000_1: or as a system property in the command line.
      attempt_201308280427_0013_m_000000_1: Exception in thread "Thread for syncLogs" java.lang.NullPointerException
      attempt_201308280427_0013_m_000000_1: at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:94)
      attempt_201308280427_0013_m_000000_1: at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
      attempt_201308280427_0013_m_000000_1: at org.apache.hadoop.mapred.Child$3.run(Child.java:153)
      attempt_201308280427_0013_m_000000_1: WARN - Error running child
      attempt_201308280427_0013_m_000000_1: java.lang.NullPointerException
      attempt_201308280427_0013_m_000000_1: at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:94)
      attempt_201308280427_0013_m_000000_1: at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
      attempt_201308280427_0013_m_000000_1: at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
      attempt_201308280427_0013_m_000000_1: at java.security.AccessController.doPrivileged(Native Method)
      attempt_201308280427_0013_m_000000_1: at javax.security.auth.Subject.doAs(Subject.java:396)
      attempt_201308280427_0013_m_000000_1: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
      attempt_201308280427_0013_m_000000_1: at org.apache.hadoop.mapred.Child.main(Child.java:260)
      attempt_201308280427_0013_m_000000_1: INFO - Runnning cleanup for the task
      INFO - Task Id : attempt_201308280427_0013_m_000000_2, Status : FAILED
      java.lang.NullPointerException
              at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:94)
              at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
              at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
              at org.apache.hadoop.mapred.Child.main(Child.java:260)

      attempt_201308280427_0013_m_000000_2: WARNING: The file terrier.properties was not found at location /etc/terrier.properties
      attempt_201308280427_0013_m_000000_2: Assuming the value of terrier.home from the corresponding system property.
      attempt_201308280427_0013_m_000000_2: Loading codebook...
      attempt_201308280427_0013_m_000000_2: codebookURL:/usr/local/hadoop_data/mapred_local/taskTracker/hadoop/jobcache/job_201308280427_0013/attempt_201308280427_0013_m_000000_2/work/tmp/imageterrier4985722866117187795.cluster
      attempt_201308280427_0013_m_000000_2: Please ensure that the property terrier.home
      attempt_201308280427_0013_m_000000_2: is specified in the file terrier.properties,
      attempt_201308280427_0013_m_000000_2: or as a system property in the command line.
      attempt_201308280427_0013_m_000000_2: Exception in thread "Thread for syncLogs" java.lang.NullPointerException
      attempt_201308280427_0013_m_000000_2: at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:94)
      attempt_201308280427_0013_m_000000_2: at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
      attempt_201308280427_0013_m_000000_2: at org.apache.hadoop.mapred.Child$3.run(Child.java:153)
      attempt_201308280427_0013_m_000000_2: WARN - Error running child
      attempt_201308280427_0013_m_000000_2: java.lang.NullPointerException
      attempt_201308280427_0013_m_000000_2: at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:94)
      attempt_201308280427_0013_m_000000_2: at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
      attempt_201308280427_0013_m_000000_2: at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
      attempt_201308280427_0013_m_000000_2: at java.security.AccessController.doPrivileged(Native Method)
      attempt_201308280427_0013_m_000000_2: at javax.security.auth.Subject.doAs(Subject.java:396)
      attempt_201308280427_0013_m_000000_2: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
      attempt_201308280427_0013_m_000000_2: at org.apache.hadoop.mapred.Child.main(Child.java:260)
      attempt_201308280427_0013_m_000000_2: INFO - Runnning cleanup for the task
      INFO - Job complete: job_201308280427_0013
      INFO - Counters: 7
      INFO - Job Counters
      INFO - SLOTS_MILLIS_MAPS=341295
      INFO - Total time spent by all reduces waiting after reserving slots (ms)=0
      INFO - Total time spent by all maps waiting after reserving slots (ms)=0
      INFO - Launched map tasks=4
      INFO - Data-local map tasks=4
      INFO - SLOTS_MILLIS_REDUCES=0
      INFO - Failed map tasks=1
      Time Taken = 401 seconds

      ---------------------
      Thanks for help. BTW, should I put it here or on the issue tracker.

        Attachments

          Issue Links

            Activity

            Hide
            craigm Craig Macdonald added a comment -

            Hi winning,

            Can you apply the patch at http://terrier.org/issues/browse/TR-201, recompile and re-try? I think this is a duplicate. Let me know.

            Many thanks

            Craig

            Show
            craigm Craig Macdonald added a comment - Hi winning, Can you apply the patch at http://terrier.org/issues/browse/TR-201 , recompile and re-try? I think this is a duplicate. Let me know. Many thanks Craig
            Hide
            atune winning added a comment - - edited

            Hi Craig ,
            excuse me.i just export my project from eclipse,and copy my project to Redhat( where hadoop exists)and run it. I just click the link you gave me but it's hard for me to know how to do. could you please tell me how to use the patch and recompile my project in detail?
            thank you very much.

            Show
            atune winning added a comment - - edited Hi Craig , excuse me.i just export my project from eclipse,and copy my project to Redhat( where hadoop exists)and run it. I just click the link you gave me but it's hard for me to know how to do. could you please tell me how to use the patch and recompile my project in detail? thank you very much.
            Hide
            craigm Craig Macdonald added a comment -

            The patch is to be applied to Terrier folder, not to your own application. Use the patch command on *nix, then recompile using "ant jar".

            Show
            craigm Craig Macdonald added a comment - The patch is to be applied to Terrier folder, not to your own application. Use the patch command on *nix, then recompile using "ant jar".
            Hide
            atune winning added a comment -

            Terrier-3.5.0.jar is in my Maven Dependencies.So I need to download Terrier-3.5.0-src and recompiled it. right?

            Show
            atune winning added a comment - Terrier-3.5.0.jar is in my Maven Dependencies.So I need to download Terrier-3.5.0-src and recompiled it. right?
            Hide
            atune winning added a comment -

            i fix it as you told me. thanks a lot,Craig.

            Show
            atune winning added a comment - i fix it as you told me. thanks a lot,Craig.

              People

              • Assignee:
                craigm Craig Macdonald
                Reporter:
                atune winning
              • Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: