Terrier Users :  Terrier Forum terrier.org
General discussion about using/developing applications using Terrier 
Terrier 3.5 with Hadoop 1.0.3
Posted by: sergg ()
Date: July 23, 2012 07:26AM

Hi,

I am trying to run Terrier (v3.5) jobs on Hadoop 1.0.3, but has a problems with it (looks like Terrier can find Hadoop classpath, but all settings/properties filled correctly).
Do you know is it possible or not?
Any ideas how to do that?

Hadoop is deployed as rpm-package.

Options: ReplyQuote
Re: Terrier 3.5 with Hadoop 1.0.3
Posted by: craigm ()
Date: July 23, 2012 11:21AM

Can you post the error message?

Craig

Options: ReplyQuote
Re: Terrier 3.5 with Hadoop 1.0.3
Posted by: sergg ()
Date: July 23, 2012 01:23PM

Actually I want to know is anybody successed to run Terrier 3.5 with Hadoop 1.0.3 because I cant find any blog/forum posts about such success experience.

Hadoop for me works fine; Hadoop plugin and indices path defined in Terrier properties. I have this error:

Setting TERRIER_HOME to /home/hduser/terrier-3.5
log4j:ERROR Could not instantiate class [org.apache.hadoop.log.metrics.EventCounter].
java.lang.ClassNotFoundException: org.apache.hadoop.log.metrics.EventCounter
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:186)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)

Options: ReplyQuote
Re: Terrier 3.5 with Hadoop 1.0.3
Posted by: craigm ()
Date: July 23, 2012 06:41PM

Ok, just to check, did you do approximately what is discussed in:
[ir.dcs.gla.ac.uk]

Craig

Options: ReplyQuote
Re: Terrier 3.5 with Hadoop 1.0.3
Posted by: sergg ()
Date: July 24, 2012 08:15AM

Craig,

Thanks for instructions. Copying Hadoop libs to Terrier ./lib/hadoop folder cause the error that HadoopPlugin cant find apache commons.
Then I copy all Hadoop libs from Terrier ./lib/hadoop to ./lib/ folder, old problem dissapears, but I obtain new one. Looks like it still wait for old Hadoop libraries. Any ideas?

org.terrier.utility.io.WrappedIOException: Cannot HadoopUtility.makeTerrierJob
at org.terrier.utility.io.HadoopUtility.makeTerrierJob(HadoopUtility.java:176)
...
Caused by: java.io.FileNotFoundException: File file:/home/hduser/terrier-3.5/lib/hadoop0.20/*.jar does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:213)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1195)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1171)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1143)
at org.terrier.utility.io.HadoopUtility.saveClassPathToJob(HadoopUtility.java:241)
at org.terrier.utility.io.HadoopUtility.makeTerrierJob(HadoopUtility.java:174)
... 6 more

Options: ReplyQuote
Re: Terrier 3.5 with Hadoop 1.0.3
Posted by: craigm ()
Date: July 24, 2012 10:20AM

Curious. Can we go for a hadoop0.20 folder for the Hadoop jar files?

Craig

Options: ReplyQuote
Re: Terrier 3.5 with Hadoop 1.0.3
Posted by: sergg ()
Date: July 24, 2012 11:57AM

Craig,

Cool idea. Now it works smiling smiley
Actually for the next Terrier release it will be better if Hadoop libraries will be placed not in ./lib/hadoop0.20 but in ./lib/hadoop. So there are no version dependency.

Thanks for your help!

Options: ReplyQuote
Re: Terrier 3.5 with Hadoop 1.0.3
Posted by: craigm ()
Date: July 24, 2012 01:14PM

I have found the issue: bin/anyclass.sh includes $TERRIER_LIB/hadoop0.20/*.jar. The exception above occurred because $TERRIER_LIB/hadoop0.20/*.jar did not match anything, and hence was passed verbatim into the classpath.

Good idea to rename the directory. See [terrier.org]

Craig



Edited 2 time(s). Last edit at 07/24/2012 01:16PM by craigm.

Options: ReplyQuote
Re: Terrier 3.5 with Hadoop 1.0.3
Posted by: sergg ()
Date: July 24, 2012 01:36PM

Craig,

Thank you very much!

Options: ReplyQuote


Sorry, only registered users may post in this forum.
This forum powered by Phorum.