Terrier Users :  Terrier Forum terrier.org
General discussion about using/developing applications using Terrier 
Running Terrier 3.5 on Hadoop
Posted by: ddesani ()
Date: July 13, 2012 11:35PM

Hi Guys,

I'm trying to run Terrier 3.5 on a small Hadoop Cluster in my college.

Here is my configuration:
Hadoop version: 0.20.203.0
Terrier version: 3.5
Nodes: 32
Sample dataset: [boston.lti.cs.cmu.edu]

----------------------------------------------------
$ cat etc/collection.spec
#add the files to index
hdfs://.../collections/ClueWeb09/ClueWeb09_Chinese_Sample.warc
hdfs://.../collections/ClueWeb09/ClueWeb09_English_Sample.warc
hdfs://.../collections/ClueWeb09/ClueWeb09_Spanish_Sample.warc
----------------------------------------------------

----------------------------------------------------
$ cat etc/terrier.properties
#directory names
terrier.home=/mnt/terrier-3.5

#default controls for query expansion
querying.postprocesses.order=QueryExpansion
querying.postprocesses.controls=qe:QueryExpansion
#default controls for the web-based interface. SimpleDecorate
#is the simplest metadata decorator. For more control, see Decorate.
querying.postfilters.order=SimpleDecorate,SiteFilter,Scope
querying.postfilters.controls=decorateconfused smileyimpleDecorate,siteconfused smileyiteFilter,scopeconfused smileycope

#default and allowed controls
querying.default.controls=start:0,end:999
querying.allowed.controls=c,scope,qe,qemodel,start,end,site,scope

#document tags specification
#for processing the contents of
#the documents, ignoring DOCHDR
TrecDocTags.doctag=DOC
TrecDocTags.idtag=DOCNO
TrecDocTags.skip=DOCHDR
#set to true if the tags can be of various case
TrecDocTags.casesensitive=false

#query tags specification
TrecQueryTags.doctag=TOP
TrecQueryTags.idtag=NUM
TrecQueryTags.process=TOP,NUM,TITLE
TrecQueryTags.skip=DESC,NARR

#stop-words file
stopwords.filename=stopword-list.txt

#the processing stages a term goes through
termpipelines=Stopwords,PorterStemmer

trec.collection.class=WARC018Collection
indexer.meta.forward.keys=docno,url
indexer.meta.forward.keylens=26,256
indexer.meta.reverse.keys=docno

#using Hadoop MapReduce, with/without HOD
terrier.index.path=hdfs://.../collections/indices

#terrier.plugins=org.terrier.utility.io.HadoopPlugin
----------------------------------------------------

----------------------------------------------------
$ cat bin/terrier-env.sh
#You can use this file to setup any additional environment variables used by anyclass.sh

#Extra entries can be added to the classpath
#CLASSPATH=

#For example, to use Hadoop support in Terrier, you need to add
#the Hadoop conf/ folder to the classpath
CLASSPATH=/mnt/hadoop-0.20.203.0/conf
----------------------------------------------------

Here is the output after trying to execute Terrier with "-H" option:
----------------------------------------------------
$ bin/trec_terrier.sh -i -H
Setting TERRIER_HOME to /mnt/terrier-3.5
log4j:ERROR Could not instantiate class [org.apache.hadoop.log.metrics.EventCounter].
java.lang.ClassNotFoundException: org.apache.hadoop.log.metrics.EventCounter
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:186)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:471)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:125)
at org.apache.log4j.Logger.getLogger(Logger.java:118)
at org.terrier.applications.TrecTerrier.<clinit>(TrecTerrier.java:85)
log4j:ERROR Could not instantiate appender named "EventCounter".
INFO - Term-partitioned Mode, 26 reducers creating one inverted index.
org.terrier.utility.io.WrappedIOException: Cannot HadoopUtility.makeTerrierJob
org.terrier.utility.io.WrappedIOException: Cannot HadoopUtility.makeTerrierJob
at org.terrier.utility.io.HadoopUtility.makeTerrierJob(HadoopUtility.java:176)
at org.terrier.utility.io.HadoopPlugin$JobFactory.makeTerrierJob(HadoopPlugin.java:122)
at org.terrier.utility.io.HadoopPlugin$DirectJobFactory.newJob(HadoopPlugin.java:137)
at org.terrier.applications.HadoopIndexing.main(HadoopIndexing.java:160)
at org.terrier.applications.TrecTerrier.run(TrecTerrier.java:371)
at org.terrier.applications.TrecTerrier.applyOptions(TrecTerrier.java:564)
at org.terrier.applications.TrecTerrier.main(TrecTerrier.java:235)
Caused by: java.io.IOException: Call to c-3/10.22.1.3:54310 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:110)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:211)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:174)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1448)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1476)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:197)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:96)
at org.terrier.utility.io.HadoopUtility.saveApplicationSetupToJob(HadoopUtility.java:299)
at org.terrier.utility.io.HadoopUtility.makeTerrierJob(HadoopUtility.java:173)
... 6 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
----------------------------------------------------

Seems terrier cannot create and submit a job. Do you guys have any idea about what is happening?
Please, let me know if any extra info is required.

Thanks!

Options: ReplyQuote
Re: Running Terrier 3.5 on Hadoop
Posted by: craigm ()
Date: July 18, 2012 11:48AM

This looks like a mismatch of hadoop version, I think.

Craig

Options: ReplyQuote
Re: Running Terrier 3.5 on Hadoop
Posted by: ddesani ()
Date: July 18, 2012 08:29PM

Thanks for you reply Craig!

I've tested with hadoop-0.20.205.0 version and unfortunately got the same error.
I noticed that a warning was added into hadoop-hadoop-namenode-debian.log:

----
...
2012-07-18 16:23:16,225 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 172.16.22.128:56131 got version 3 expected version 4
----

Do you guys suggest any specific version to execute terrier 3.5?
[www.us.apache.org]

Thanks!

- Diego

Options: ReplyQuote
Re: Running Terrier 3.5 on Hadoop
Posted by: craigm ()
Date: July 19, 2012 11:05AM

I think you can change the Hadoop jar files that come with Terrier, and the corresponding lib/hadoop0.20 folder for the ones from Hadoop, and all should work fine (in theory).

Craig

UPDATE: I added a wiki page describing this:
[ir.dcs.gla.ac.uk]



Edited 1 time(s). Last edit at 07/23/2012 11:27AM by craigm.

Options: ReplyQuote
Re: Running Terrier 3.5 on Hadoop
Posted by: ddesani ()
Date: July 19, 2012 08:50PM

=)

The job now has started after updating jar files as you suggested, but unfortunately it throws an error related to stopwords file. I'm using the one that comes from terrier package and it's not empty.

Job status:
----------------
User: hadoop
Job Name: terrierIndexing
Job File: hdfs://c-3:54310/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201207191341_0002/job.xml
Submit Host: c-3
Submit Host Address: 10.22.1.3
Job-ACLs: All users are allowed
Job Setup: Successful
Status: Failed
Failure Info:NA
Started at: Thu Jul 19 15:28:32 EDT 2012
Failed at: Thu Jul 19 15:29:37 EDT 2012
Failed in: 1mins, 4sec
Job Cleanup: Successful
----------------

I tried both:
1) stopwords file on linux filesystem:
#stop-words file
stopwords.filename=/mnt/terrier-3.5/share/stopword-list.txt

2) stopwords on HDFS
#stop-words file
stopwords.filename=hdfs://c-3:54310/user/hadoop/collections/stopword-list.txt

They have 644 permission:
----------------
[hadoop@c-3 terrier-3.5]$ ls -l /mnt/terrier-3.5/share/stopword-list.txt
-rw-r--r-- 1 hadoop hadoop 5230 Jun 16 2011 /mnt/terrier-3.5/share/stopword-list.txt
[hadoop@c-3 terrier-3.5]$ hadoop dfs -ls collections/stopword-list.txt
Found 1 items
-rw-r--r-- 3 hadoop supergroup 5230 2012-07-19 12:58 /user/hadoop/collections/stopword-list.txt
[hadoop@c-3 terrier-3.5]$
----------------

Any special way to set this?

Log:
----------------
$ bin/trec_terrier.sh -i -H
Setting TERRIER_HOME to /mnt/terrier-3.5
INFO - Term-partitioned Mode, 26 reducers creating one inverted index.
INFO - Copying terrier share/ directory (/mnt/terrier-3.5/share) to shared storage area (hdfs://c-3:54310/tmp/1650971759-terrier.share)
INFO - Copying classpath to job
WARN - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
WARN - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
INFO - Allocating 3 files across 2 map tasks
INFO - Running job: job_201207191341_0002
INFO - map 0% reduce 0%
INFO - Task Id : attempt_201207191341_0002_m_000000_0, Status : FAILED
java.lang.NullPointerException
at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)

attempt_201207191341_0002_m_000000_0: WARNING: The file terrier.properties was not found at location /mnt/terrier-3.5/etc/terrier.properties
attempt_201207191341_0002_m_000000_0: Assuming the value of terrier.home from the corresponding system property.
attempt_201207191341_0002_m_000000_0: INFO - numReduceTasks: 26
attempt_201207191341_0002_m_000000_0: INFO - io.sort.mb = 100
attempt_201207191341_0002_m_000000_0: INFO - data buffer = 79691776/99614720
attempt_201207191341_0002_m_000000_0: INFO - record buffer = 262144/327680
attempt_201207191341_0002_m_000000_0: INFO - Reloading Application Setup
attempt_201207191341_0002_m_000000_0: INFO - Checking memory usage every 20 maxDocPerFlush=0
attempt_201207191341_0002_m_000000_0: ERROR - Errror: Input/Output Exception while reading stopword list (/mnt/terrier-3.5/share/stopword-list.txt) : Stack trace follows.
attempt_201207191341_0002_m_000000_0: java.io.FileNotFoundException: /mnt/terrier-3.5/share/stopword-list.txt (No such file or directory)
attempt_201207191341_0002_m_000000_0: at java.io.FileInputStream.open(Native Method)
attempt_201207191341_0002_m_000000_0: at java.io.FileInputStream.<init>(FileInputStream.java:137)
attempt_201207191341_0002_m_000000_0: at java.io.FileInputStream.<init>(FileInputStream.java:96)
attempt_201207191341_0002_m_000000_0: at org.terrier.utility.io.LocalFileSystem.openFileStream(LocalFileSystem.java:104)
attempt_201207191341_0002_m_000000_0: at org.terrier.utility.Files.openFile(Files.java:300)
attempt_201207191341_0002_m_000000_0: at org.terrier.utility.Files.openFileReader(Files.java:608)
attempt_201207191341_0002_m_000000_0: at org.terrier.utility.Files.openFileReader(Files.java:596)
attempt_201207191341_0002_m_000000_0: at org.terrier.terms.Stopwords.loadStopwordsList(Stopwords.java:128)
attempt_201207191341_0002_m_000000_0: at org.terrier.terms.Stopwords.<init>(Stopwords.java:86)
attempt_201207191341_0002_m_000000_0: at org.terrier.terms.Stopwords.<init>(Stopwords.java:69)
attempt_201207191341_0002_m_000000_0: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
attempt_201207191341_0002_m_000000_0: at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
attempt_201207191341_0002_m_000000_0: at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
attempt_201207191341_0002_m_000000_0: at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
attempt_201207191341_0002_m_000000_0: at org.terrier.indexing.Indexer.load_pipeline(Indexer.java:294)
attempt_201207191341_0002_m_000000_0: at org.terrier.indexing.Indexer.init(Indexer.java:191)
attempt_201207191341_0002_m_000000_0: at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configureMap(Hadoop_BasicSinglePassIndexer.java:277)
attempt_201207191341_0002_m_000000_0: at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configure(Hadoop_BasicSinglePassIndexer.java:230)
attempt_201207191341_0002_m_000000_0: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
attempt_201207191341_0002_m_000000_0: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
attempt_201207191341_0002_m_000000_0: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
attempt_201207191341_0002_m_000000_0: at java.lang.reflect.Method.invoke(Method.java:616)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
attempt_201207191341_0002_m_000000_0: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
attempt_201207191341_0002_m_000000_0: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
attempt_201207191341_0002_m_000000_0: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
attempt_201207191341_0002_m_000000_0: at java.lang.reflect.Method.invoke(Method.java:616)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:431)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
attempt_201207191341_0002_m_000000_0: at java.security.AccessController.doPrivileged(Native Method)
attempt_201207191341_0002_m_000000_0: at javax.security.auth.Subject.doAs(Subject.java:416)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201207191341_0002_m_000000_0: ERROR - Error: Empty stopwords file was used (/mnt/terrier-3.5/share/stopword-list.txt)
attempt_201207191341_0002_m_000000_0: WARN - Error running child
attempt_201207191341_0002_m_000000_0: java.lang.NullPointerException
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
attempt_201207191341_0002_m_000000_0: at java.security.AccessController.doPrivileged(Native Method)
attempt_201207191341_0002_m_000000_0: at javax.security.auth.Subject.doAs(Subject.java:416)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
attempt_201207191341_0002_m_000000_0: at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201207191341_0002_m_000000_0: INFO - Runnning cleanup for the task
INFO - Task Id : attempt_201207191341_0002_m_000001_0, Status : FAILED
java.lang.NullPointerException
at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)

attempt_201207191341_0002_m_000001_0: WARNING: The file terrier.properties was not found at location /mnt/terrier-3.5/etc/terrier.properties
attempt_201207191341_0002_m_000001_0: Assuming the value of terrier.home from the corresponding system property.
attempt_201207191341_0002_m_000001_0: INFO - numReduceTasks: 26
attempt_201207191341_0002_m_000001_0: INFO - io.sort.mb = 100
attempt_201207191341_0002_m_000001_0: INFO - data buffer = 79691776/99614720
attempt_201207191341_0002_m_000001_0: INFO - record buffer = 262144/327680
attempt_201207191341_0002_m_000001_0: INFO - Reloading Application Setup
attempt_201207191341_0002_m_000001_0: INFO - Checking memory usage every 20 maxDocPerFlush=0
attempt_201207191341_0002_m_000001_0: ERROR - Errror: Input/Output Exception while reading stopword list (/mnt/terrier-3.5/share/stopword-list.txt) : Stack trace follows.
attempt_201207191341_0002_m_000001_0: java.io.FileNotFoundException: /mnt/terrier-3.5/share/stopword-list.txt (No such file or directory)
attempt_201207191341_0002_m_000001_0: at java.io.FileInputStream.open(Native Method)
attempt_201207191341_0002_m_000001_0: at java.io.FileInputStream.<init>(FileInputStream.java:137)
attempt_201207191341_0002_m_000001_0: at java.io.FileInputStream.<init>(FileInputStream.java:96)
attempt_201207191341_0002_m_000001_0: at org.terrier.utility.io.LocalFileSystem.openFileStream(LocalFileSystem.java:104)
attempt_201207191341_0002_m_000001_0: at org.terrier.utility.Files.openFile(Files.java:300)
attempt_201207191341_0002_m_000001_0: at org.terrier.utility.Files.openFileReader(Files.java:608)
attempt_201207191341_0002_m_000001_0: at org.terrier.utility.Files.openFileReader(Files.java:596)
attempt_201207191341_0002_m_000001_0: at org.terrier.terms.Stopwords.loadStopwordsList(Stopwords.java:128)
attempt_201207191341_0002_m_000001_0: at org.terrier.terms.Stopwords.<init>(Stopwords.java:86)
attempt_201207191341_0002_m_000001_0: at org.terrier.terms.Stopwords.<init>(Stopwords.java:69)
attempt_201207191341_0002_m_000001_0: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
attempt_201207191341_0002_m_000001_0: at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
attempt_201207191341_0002_m_000001_0: at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
attempt_201207191341_0002_m_000001_0: at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
attempt_201207191341_0002_m_000001_0: at org.terrier.indexing.Indexer.load_pipeline(Indexer.java:294)
attempt_201207191341_0002_m_000001_0: at org.terrier.indexing.Indexer.init(Indexer.java:191)
attempt_201207191341_0002_m_000001_0: at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configureMap(Hadoop_BasicSinglePassIndexer.java:277)
attempt_201207191341_0002_m_000001_0: at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configure(Hadoop_BasicSinglePassIndexer.java:230)
attempt_201207191341_0002_m_000001_0: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
attempt_201207191341_0002_m_000001_0: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
attempt_201207191341_0002_m_000001_0: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
attempt_201207191341_0002_m_000001_0: at java.lang.reflect.Method.invoke(Method.java:616)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
attempt_201207191341_0002_m_000001_0: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
attempt_201207191341_0002_m_000001_0: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
attempt_201207191341_0002_m_000001_0: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
attempt_201207191341_0002_m_000001_0: at java.lang.reflect.Method.invoke(Method.java:616)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:431)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
attempt_201207191341_0002_m_000001_0: at java.security.AccessController.doPrivileged(Native Method)
attempt_201207191341_0002_m_000001_0: at javax.security.auth.Subject.doAs(Subject.java:416)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201207191341_0002_m_000001_0: ERROR - Error: Empty stopwords file was used (/mnt/terrier-3.5/share/stopword-list.txt)
attempt_201207191341_0002_m_000001_0: WARN - Error running child
attempt_201207191341_0002_m_000001_0: java.lang.NullPointerException
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
attempt_201207191341_0002_m_000001_0: at java.security.AccessController.doPrivileged(Native Method)
attempt_201207191341_0002_m_000001_0: at javax.security.auth.Subject.doAs(Subject.java:416)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
attempt_201207191341_0002_m_000001_0: at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201207191341_0002_m_000001_0: INFO - Runnning cleanup for the task
INFO - Task Id : attempt_201207191341_0002_m_000000_1, Status : FAILED
java.lang.NullPointerException
at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)

attempt_201207191341_0002_m_000000_1: WARNING: The file terrier.properties was not found at location /mnt/terrier-3.5/etc/terrier.properties
attempt_201207191341_0002_m_000000_1: Assuming the value of terrier.home from the corresponding system property.
attempt_201207191341_0002_m_000000_1: INFO - numReduceTasks: 26
attempt_201207191341_0002_m_000000_1: INFO - io.sort.mb = 100
attempt_201207191341_0002_m_000000_1: INFO - data buffer = 79691776/99614720
attempt_201207191341_0002_m_000000_1: INFO - record buffer = 262144/327680
attempt_201207191341_0002_m_000000_1: INFO - Reloading Application Setup
attempt_201207191341_0002_m_000000_1: INFO - Checking memory usage every 20 maxDocPerFlush=0
attempt_201207191341_0002_m_000000_1: ERROR - Errror: Input/Output Exception while reading stopword list (/mnt/terrier-3.5/share/stopword-list.txt) : Stack trace follows.
attempt_201207191341_0002_m_000000_1: java.io.FileNotFoundException: /mnt/terrier-3.5/share/stopword-list.txt (No such file or directory)
attempt_201207191341_0002_m_000000_1: at java.io.FileInputStream.open(Native Method)
attempt_201207191341_0002_m_000000_1: at java.io.FileInputStream.<init>(FileInputStream.java:137)
attempt_201207191341_0002_m_000000_1: at java.io.FileInputStream.<init>(FileInputStream.java:96)
attempt_201207191341_0002_m_000000_1: at org.terrier.utility.io.LocalFileSystem.openFileStream(LocalFileSystem.java:104)
attempt_201207191341_0002_m_000000_1: at org.terrier.utility.Files.openFile(Files.java:300)
attempt_201207191341_0002_m_000000_1: at org.terrier.utility.Files.openFileReader(Files.java:608)
attempt_201207191341_0002_m_000000_1: at org.terrier.utility.Files.openFileReader(Files.java:596)
attempt_201207191341_0002_m_000000_1: at org.terrier.terms.Stopwords.loadStopwordsList(Stopwords.java:128)
attempt_201207191341_0002_m_000000_1: at org.terrier.terms.Stopwords.<init>(Stopwords.java:86)
attempt_201207191341_0002_m_000000_1: at org.terrier.terms.Stopwords.<init>(Stopwords.java:69)
attempt_201207191341_0002_m_000000_1: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
attempt_201207191341_0002_m_000000_1: at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
attempt_201207191341_0002_m_000000_1: at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
attempt_201207191341_0002_m_000000_1: at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
attempt_201207191341_0002_m_000000_1: at org.terrier.indexing.Indexer.load_pipeline(Indexer.java:294)
attempt_201207191341_0002_m_000000_1: at org.terrier.indexing.Indexer.init(Indexer.java:191)
attempt_201207191341_0002_m_000000_1: at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configureMap(Hadoop_BasicSinglePassIndexer.java:277)
attempt_201207191341_0002_m_000000_1: at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configure(Hadoop_BasicSinglePassIndexer.java:230)
attempt_201207191341_0002_m_000000_1: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
attempt_201207191341_0002_m_000000_1: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
attempt_201207191341_0002_m_000000_1: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
attempt_201207191341_0002_m_000000_1: at java.lang.reflect.Method.invoke(Method.java:616)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
attempt_201207191341_0002_m_000000_1: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
attempt_201207191341_0002_m_000000_1: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
attempt_201207191341_0002_m_000000_1: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
attempt_201207191341_0002_m_000000_1: at java.lang.reflect.Method.invoke(Method.java:616)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:431)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
attempt_201207191341_0002_m_000000_1: at java.security.AccessController.doPrivileged(Native Method)
attempt_201207191341_0002_m_000000_1: at javax.security.auth.Subject.doAs(Subject.java:416)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201207191341_0002_m_000000_1: ERROR - Error: Empty stopwords file was used (/mnt/terrier-3.5/share/stopword-list.txt)
attempt_201207191341_0002_m_000000_1: WARN - Error running child
attempt_201207191341_0002_m_000000_1: java.lang.NullPointerException
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
attempt_201207191341_0002_m_000000_1: at java.security.AccessController.doPrivileged(Native Method)
attempt_201207191341_0002_m_000000_1: at javax.security.auth.Subject.doAs(Subject.java:416)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
attempt_201207191341_0002_m_000000_1: at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201207191341_0002_m_000000_1: INFO - Runnning cleanup for the task
INFO - Task Id : attempt_201207191341_0002_m_000001_1, Status : FAILED
java.lang.NullPointerException
at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)

attempt_201207191341_0002_m_000001_1: WARNING: The file terrier.properties was not found at location /mnt/terrier-3.5/etc/terrier.properties
attempt_201207191341_0002_m_000001_1: Assuming the value of terrier.home from the corresponding system property.
attempt_201207191341_0002_m_000001_1: INFO - numReduceTasks: 26
attempt_201207191341_0002_m_000001_1: INFO - io.sort.mb = 100
attempt_201207191341_0002_m_000001_1: INFO - data buffer = 79691776/99614720
attempt_201207191341_0002_m_000001_1: INFO - record buffer = 262144/327680
attempt_201207191341_0002_m_000001_1: INFO - Reloading Application Setup
attempt_201207191341_0002_m_000001_1: INFO - Checking memory usage every 20 maxDocPerFlush=0
attempt_201207191341_0002_m_000001_1: ERROR - Errror: Input/Output Exception while reading stopword list (/mnt/terrier-3.5/share/stopword-list.txt) : Stack trace follows.
attempt_201207191341_0002_m_000001_1: java.io.FileNotFoundException: /mnt/terrier-3.5/share/stopword-list.txt (No such file or directory)
attempt_201207191341_0002_m_000001_1: at java.io.FileInputStream.open(Native Method)
attempt_201207191341_0002_m_000001_1: at java.io.FileInputStream.<init>(FileInputStream.java:137)
attempt_201207191341_0002_m_000001_1: at java.io.FileInputStream.<init>(FileInputStream.java:96)
attempt_201207191341_0002_m_000001_1: at org.terrier.utility.io.LocalFileSystem.openFileStream(LocalFileSystem.java:104)
attempt_201207191341_0002_m_000001_1: at org.terrier.utility.Files.openFile(Files.java:300)
attempt_201207191341_0002_m_000001_1: at org.terrier.utility.Files.openFileReader(Files.java:608)
attempt_201207191341_0002_m_000001_1: at org.terrier.utility.Files.openFileReader(Files.java:596)
attempt_201207191341_0002_m_000001_1: at org.terrier.terms.Stopwords.loadStopwordsList(Stopwords.java:128)
attempt_201207191341_0002_m_000001_1: at org.terrier.terms.Stopwords.<init>(Stopwords.java:86)
attempt_201207191341_0002_m_000001_1: at org.terrier.terms.Stopwords.<init>(Stopwords.java:69)
attempt_201207191341_0002_m_000001_1: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
attempt_201207191341_0002_m_000001_1: at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
attempt_201207191341_0002_m_000001_1: at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
attempt_201207191341_0002_m_000001_1: at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
attempt_201207191341_0002_m_000001_1: at org.terrier.indexing.Indexer.load_pipeline(Indexer.java:294)
attempt_201207191341_0002_m_000001_1: at org.terrier.indexing.Indexer.init(Indexer.java:191)
attempt_201207191341_0002_m_000001_1: at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configureMap(Hadoop_BasicSinglePassIndexer.java:277)
attempt_201207191341_0002_m_000001_1: at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configure(Hadoop_BasicSinglePassIndexer.java:230)
attempt_201207191341_0002_m_000001_1: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
attempt_201207191341_0002_m_000001_1: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
attempt_201207191341_0002_m_000001_1: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
attempt_201207191341_0002_m_000001_1: at java.lang.reflect.Method.invoke(Method.java:616)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
attempt_201207191341_0002_m_000001_1: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
attempt_201207191341_0002_m_000001_1: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
attempt_201207191341_0002_m_000001_1: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
attempt_201207191341_0002_m_000001_1: at java.lang.reflect.Method.invoke(Method.java:616)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:431)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
attempt_201207191341_0002_m_000001_1: at java.security.AccessController.doPrivileged(Native Method)
attempt_201207191341_0002_m_000001_1: at javax.security.auth.Subject.doAs(Subject.java:416)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201207191341_0002_m_000001_1: ERROR - Error: Empty stopwords file was used (/mnt/terrier-3.5/share/stopword-list.txt)
attempt_201207191341_0002_m_000001_1: WARN - Error running child
attempt_201207191341_0002_m_000001_1: java.lang.NullPointerException
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
attempt_201207191341_0002_m_000001_1: at java.security.AccessController.doPrivileged(Native Method)
attempt_201207191341_0002_m_000001_1: at javax.security.auth.Subject.doAs(Subject.java:416)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
attempt_201207191341_0002_m_000001_1: at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201207191341_0002_m_000001_1: INFO - Runnning cleanup for the task
INFO - Task Id : attempt_201207191341_0002_m_000000_2, Status : FAILED
java.lang.NullPointerException
at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)

attempt_201207191341_0002_m_000000_2: WARNING: The file terrier.properties was not found at location /mnt/terrier-3.5/etc/terrier.properties
attempt_201207191341_0002_m_000000_2: Assuming the value of terrier.home from the corresponding system property.
attempt_201207191341_0002_m_000000_2: INFO - numReduceTasks: 26
attempt_201207191341_0002_m_000000_2: INFO - io.sort.mb = 100
attempt_201207191341_0002_m_000000_2: INFO - data buffer = 79691776/99614720
attempt_201207191341_0002_m_000000_2: INFO - record buffer = 262144/327680
attempt_201207191341_0002_m_000000_2: INFO - Reloading Application Setup
attempt_201207191341_0002_m_000000_2: INFO - Checking memory usage every 20 maxDocPerFlush=0
attempt_201207191341_0002_m_000000_2: ERROR - Errror: Input/Output Exception while reading stopword list (/mnt/terrier-3.5/share/stopword-list.txt) : Stack trace follows.
attempt_201207191341_0002_m_000000_2: java.io.FileNotFoundException: /mnt/terrier-3.5/share/stopword-list.txt (No such file or directory)
attempt_201207191341_0002_m_000000_2: at java.io.FileInputStream.open(Native Method)
attempt_201207191341_0002_m_000000_2: at java.io.FileInputStream.<init>(FileInputStream.java:137)
attempt_201207191341_0002_m_000000_2: at java.io.FileInputStream.<init>(FileInputStream.java:96)
attempt_201207191341_0002_m_000000_2: at org.terrier.utility.io.LocalFileSystem.openFileStream(LocalFileSystem.java:104)
attempt_201207191341_0002_m_000000_2: at org.terrier.utility.Files.openFile(Files.java:300)
attempt_201207191341_0002_m_000000_2: at org.terrier.utility.Files.openFileReader(Files.java:608)
attempt_201207191341_0002_m_000000_2: at org.terrier.utility.Files.openFileReader(Files.java:596)
attempt_201207191341_0002_m_000000_2: at org.terrier.terms.Stopwords.loadStopwordsList(Stopwords.java:128)
attempt_201207191341_0002_m_000000_2: at org.terrier.terms.Stopwords.<init>(Stopwords.java:86)
attempt_201207191341_0002_m_000000_2: at org.terrier.terms.Stopwords.<init>(Stopwords.java:69)
attempt_201207191341_0002_m_000000_2: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
attempt_201207191341_0002_m_000000_2: at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
attempt_201207191341_0002_m_000000_2: at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
attempt_201207191341_0002_m_000000_2: at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
attempt_201207191341_0002_m_000000_2: at org.terrier.indexing.Indexer.load_pipeline(Indexer.java:294)
attempt_201207191341_0002_m_000000_2: at org.terrier.indexing.Indexer.init(Indexer.java:191)
attempt_201207191341_0002_m_000000_2: at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configureMap(Hadoop_BasicSinglePassIndexer.java:277)
attempt_201207191341_0002_m_000000_2: at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configure(Hadoop_BasicSinglePassIndexer.java:230)
attempt_201207191341_0002_m_000000_2: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
attempt_201207191341_0002_m_000000_2: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
attempt_201207191341_0002_m_000000_2: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
attempt_201207191341_0002_m_000000_2: at java.lang.reflect.Method.invoke(Method.java:616)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
attempt_201207191341_0002_m_000000_2: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
attempt_201207191341_0002_m_000000_2: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
attempt_201207191341_0002_m_000000_2: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
attempt_201207191341_0002_m_000000_2: at java.lang.reflect.Method.invoke(Method.java:616)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:431)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
attempt_201207191341_0002_m_000000_2: at java.security.AccessController.doPrivileged(Native Method)
attempt_201207191341_0002_m_000000_2: at javax.security.auth.Subject.doAs(Subject.java:416)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201207191341_0002_m_000000_2: ERROR - Error: Empty stopwords file was used (/mnt/terrier-3.5/share/stopword-list.txt)
attempt_201207191341_0002_m_000000_2: WARN - Error running child
attempt_201207191341_0002_m_000000_2: java.lang.NullPointerException
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
attempt_201207191341_0002_m_000000_2: at java.security.AccessController.doPrivileged(Native Method)
attempt_201207191341_0002_m_000000_2: at javax.security.auth.Subject.doAs(Subject.java:416)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
attempt_201207191341_0002_m_000000_2: at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201207191341_0002_m_000000_2: INFO - Runnning cleanup for the task
INFO - Task Id : attempt_201207191341_0002_m_000001_2, Status : FAILED
java.lang.NullPointerException
at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)

attempt_201207191341_0002_m_000001_2: WARNING: The file terrier.properties was not found at location /mnt/terrier-3.5/etc/terrier.properties
attempt_201207191341_0002_m_000001_2: Assuming the value of terrier.home from the corresponding system property.
attempt_201207191341_0002_m_000001_2: INFO - numReduceTasks: 26
attempt_201207191341_0002_m_000001_2: INFO - io.sort.mb = 100
attempt_201207191341_0002_m_000001_2: INFO - data buffer = 79691776/99614720
attempt_201207191341_0002_m_000001_2: INFO - record buffer = 262144/327680
attempt_201207191341_0002_m_000001_2: INFO - Reloading Application Setup
attempt_201207191341_0002_m_000001_2: INFO - Checking memory usage every 20 maxDocPerFlush=0
attempt_201207191341_0002_m_000001_2: ERROR - Errror: Input/Output Exception while reading stopword list (/mnt/terrier-3.5/share/stopword-list.txt) : Stack trace follows.
attempt_201207191341_0002_m_000001_2: java.io.FileNotFoundException: /mnt/terrier-3.5/share/stopword-list.txt (No such file or directory)
attempt_201207191341_0002_m_000001_2: at java.io.FileInputStream.open(Native Method)
attempt_201207191341_0002_m_000001_2: at java.io.FileInputStream.<init>(FileInputStream.java:137)
attempt_201207191341_0002_m_000001_2: at java.io.FileInputStream.<init>(FileInputStream.java:96)
attempt_201207191341_0002_m_000001_2: at org.terrier.utility.io.LocalFileSystem.openFileStream(LocalFileSystem.java:104)
attempt_201207191341_0002_m_000001_2: at org.terrier.utility.Files.openFile(Files.java:300)
attempt_201207191341_0002_m_000001_2: at org.terrier.utility.Files.openFileReader(Files.java:608)
attempt_201207191341_0002_m_000001_2: at org.terrier.utility.Files.openFileReader(Files.java:596)
attempt_201207191341_0002_m_000001_2: at org.terrier.terms.Stopwords.loadStopwordsList(Stopwords.java:128)
attempt_201207191341_0002_m_000001_2: at org.terrier.terms.Stopwords.<init>(Stopwords.java:86)
attempt_201207191341_0002_m_000001_2: at org.terrier.terms.Stopwords.<init>(Stopwords.java:69)
attempt_201207191341_0002_m_000001_2: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
attempt_201207191341_0002_m_000001_2: at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
attempt_201207191341_0002_m_000001_2: at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
attempt_201207191341_0002_m_000001_2: at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
attempt_201207191341_0002_m_000001_2: at org.terrier.indexing.Indexer.load_pipeline(Indexer.java:294)
attempt_201207191341_0002_m_000001_2: at org.terrier.indexing.Indexer.init(Indexer.java:191)
attempt_201207191341_0002_m_000001_2: at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configureMap(Hadoop_BasicSinglePassIndexer.java:277)
attempt_201207191341_0002_m_000001_2: at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configure(Hadoop_BasicSinglePassIndexer.java:230)
attempt_201207191341_0002_m_000001_2: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
attempt_201207191341_0002_m_000001_2: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
attempt_201207191341_0002_m_000001_2: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
attempt_201207191341_0002_m_000001_2: at java.lang.reflect.Method.invoke(Method.java:616)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
attempt_201207191341_0002_m_000001_2: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
attempt_201207191341_0002_m_000001_2: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
attempt_201207191341_0002_m_000001_2: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
attempt_201207191341_0002_m_000001_2: at java.lang.reflect.Method.invoke(Method.java:616)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:431)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
attempt_201207191341_0002_m_000001_2: at java.security.AccessController.doPrivileged(Native Method)
attempt_201207191341_0002_m_000001_2: at javax.security.auth.Subject.doAs(Subject.java:416)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201207191341_0002_m_000001_2: ERROR - Error: Empty stopwords file was used (/mnt/terrier-3.5/share/stopword-list.txt)
attempt_201207191341_0002_m_000001_2: WARN - Error running child
attempt_201207191341_0002_m_000001_2: java.lang.NullPointerException
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
attempt_201207191341_0002_m_000001_2: at java.security.AccessController.doPrivileged(Native Method)
attempt_201207191341_0002_m_000001_2: at javax.security.auth.Subject.doAs(Subject.java:416)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
attempt_201207191341_0002_m_000001_2: at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201207191341_0002_m_000001_2: INFO - Runnning cleanup for the task
INFO - Job complete: job_201207191341_0002
INFO - Counters: 7
INFO - Job Counters
INFO - SLOTS_MILLIS_MAPS=62540
INFO - Total time spent by all reduces waiting after reserving slots (ms)=0
INFO - Total time spent by all maps waiting after reserving slots (ms)=0
INFO - Rack-local map tasks=8
INFO - Launched map tasks=8
INFO - SLOTS_MILLIS_REDUCES=0
INFO - Failed map tasks=1
INFO - Job Failed: NA
ERROR - Problem running job
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1204)
at org.terrier.applications.HadoopIndexing.main(HadoopIndexing.java:230)
at org.terrier.applications.TrecTerrier.run(TrecTerrier.java:371)
at org.terrier.applications.TrecTerrier.applyOptions(TrecTerrier.java:564)
at org.terrier.applications.TrecTerrier.main(TrecTerrier.java:235)
Time Taken = 74 seconds
Time elapsed: 74.766 seconds.
----------------

Thanks Craig!

- Diego

Options: ReplyQuote
Re: Running Terrier 3.5 on Hadoop
Posted by: craigm ()
Date: July 23, 2012 11:20AM

Can you make the path non-absolute:
i.e.
stopwords.filename=stopword-list.txt

Terrier will copy the entire share/ folder to HDFS, to ensure the stopwords list work ok. However, this seems to need an absolute path.

Craig

Options: ReplyQuote
Re: Running Terrier 3.5 on Hadoop
Posted by: ddesani ()
Date: July 25, 2012 07:43PM

Hey Craig, thanks for your reply and effort for solving our issue.

We set "stopwords.filename" property as you said and the issue remains. The file was copied to HDFS.

Stopword-list property:
---------------
[hadoop@c-3 terrier-3.5]$ cat etc/terrier.properties | grep stopword-list
#stopwords.filename=/mnt/terrier-3.5/share/stopword-list.txt
stopwords.filename=stopword-list.txt
---------------

Here is the error:
---------------
ERROR - Errror: Input/Output Exception while reading stopword list (hdfs://c-3:54310/tmp/1760696080-terrier.share/stopword-list.txt) : Stack trace follows.
java.io.FileNotFoundException: No file system for hdfs://c-3:54310/tmp/1760696080-terrier.share/stopword-list.txt
at org.terrier.utility.Files.openFile(Files.java:296)
at org.terrier.utility.Files.openFileReader(Files.java:608)
at org.terrier.utility.Files.openFileReader(Files.java:596)
at org.terrier.terms.Stopwords.loadStopwordsList(Stopwords.java:128)
at org.terrier.terms.Stopwords.<init>(Stopwords.java:86)
at org.terrier.terms.Stopwords.<init>(Stopwords.java:69)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at org.terrier.indexing.Indexer.load_pipeline(Indexer.java:294)
at org.terrier.indexing.Indexer.init(Indexer.java:191)
at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configureMap(Hadoop_BasicSinglePassIndexer.java:277)
at org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer.configure(Hadoop_BasicSinglePassIndexer.java:230)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:431)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
ERROR - Error: Empty stopwords file was used (hdfs://c-3:54310/tmp/1760696080-terrier.share/stopword-list.txt)
WARN - Error running child
java.lang.NullPointerException
at org.apache.hadoop.mapred.TaskLogAppender.flush(TaskLogAppender.java:67)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:264)
at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
INFO - Runnning cleanup for the task
---------------

The file was copied to HDFS by Terrier:
---------------
[hadoop@c-3 terrier-3.5]$ hadoop dfs -ls hdfs://c-3:54310/tmp/1760696080-terrier.share/stopword-list.txt
Found 1 items
-rw-r--r-- 3 hadoop supergroup 5230 2012-07-25 13:14 /tmp/1760696080-terrier.share/stopword-list.txt
---------------


Should Terrier use "org.terrier.utility.Files.openFile(Files.java:296)" to open a file from HDFS ?

Thanks Craig!

- D.D.

Options: ReplyQuote
Re: Running Terrier 3.5 on Hadoop
Posted by: craigm ()
Date: July 26, 2012 12:33AM

Aha, now we're getting somewhere.

Yes, the Files layer of Terrier can deal with different file systems, based on the protocol prefix (in this case "hdfs"winking smiley.

If Files says "No file system", then the HadoopPlugin might not be loaded. As per [terrier.org], do you have terrier.plugins=org.terrier.utility.io.HadoopPlugin in your terrier.properties file?

Craig

Options: ReplyQuote
Re: Running Terrier 3.5 on Hadoop
Posted by: ddesani ()
Date: July 27, 2012 02:51AM

=)

Exaclty! I commented out that line a while ago as you can see in the first post.
I think we can close this topic now, it's working.

Log:
---------------------

bin/trec_terrier.sh -i -H
Setting TERRIER_HOME to /mnt/terrier-3.5
INFO - Term-partitioned Mode, 26 reducers creating one inverted index.
INFO - Copying terrier share/ directory (/mnt/terrier-3.5/share) to shared storage area (hdfs://c-3:54310/tmp/893088278-terrier.share)
INFO - Copying classpath to job
WARN - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
WARN - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
INFO - Allocating 3 files across 2 map tasks
INFO - Running job: job_201207191341_0006
INFO - map 0% reduce 0%
INFO - map 50% reduce 0%
INFO - map 100% reduce 0%
INFO - map 100% reduce 1%
INFO - map 100% reduce 3%
INFO - map 100% reduce 13%
INFO - map 100% reduce 23%
INFO - map 100% reduce 30%
INFO - map 100% reduce 40%
INFO - map 100% reduce 42%
INFO - map 100% reduce 46%
INFO - map 100% reduce 64%
INFO - map 100% reduce 82%
INFO - map 100% reduce 87%
INFO - map 100% reduce 100%
INFO - Job complete: job_201207191341_0006
INFO - Counters: 30
INFO - Job Counters
INFO - Launched reduce tasks=26
INFO - SLOTS_MILLIS_MAPS=34770
INFO - Total time spent by all reduces waiting after reserving slots (ms)=0
INFO - Total time spent by all maps waiting after reserving slots (ms)=0
INFO - Rack-local map tasks=2
INFO - Launched map tasks=2
INFO - SLOTS_MILLIS_REDUCES=352565
INFO - File Input Format Counters
INFO - Bytes Read=0
INFO - File Output Format Counters
INFO - Bytes Written=0
INFO - FileSystemCounters
INFO - FILE_BYTES_READ=244715
INFO - HDFS_BYTES_READ=7824245
INFO - FILE_BYTES_WRITTEN=1491320
INFO - HDFS_BYTES_WRITTEN=1554571
INFO - org.terrier.indexing.hadoop.Hadoop_BasicSinglePassIndexer$Counters
INFO - INDEXED_POINTERS=47224
INFO - INDEXED_TOKENS=106675
INFO - INDEXED_DOCUMENTS=300
INFO - INDEXER_FLUSHES=17
INFO - Map-Reduce Framework
INFO - Map output materialized bytes=246854
INFO - Map input records=300
INFO - Reduce shuffle bytes=220225
INFO - Spilled Records=56280
INFO - Map output bytes=1843745
INFO - Map input bytes=-2381395
INFO - Combine input records=0
INFO - SPLIT_RAW_BYTES=571
INFO - Reduce input records=28140
INFO - Reduce input groups=15729
INFO - Combine output records=0
INFO - Reduce output records=0
INFO - Map output records=28140
INFO - Optimising structure lexicon
INFO - Optimsing lexicon with 15729 entries
INFO - All ids for structure lexicon are aligned, skipping .fsomapid file
INFO - Copying terrier share/ directory (/mnt/terrier-3.5/share) to shared storage area (hdfs://c-3:54310/tmp/-1093460883-terrier.share)
INFO - Copying classpath to job
WARN - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
WARN - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
INFO - Reloading Application Setup
DEBUG - Block size for hdfs://c-3:54310/user/hadoop/collections/indices/data.meta.zdata is 67108864
DEBUG - Got last split: MetaIndexSplit: BlockSize=24758 startAt=0 hosts=c-18,c-12,c-20 ids=[0,299]
DEBUG - Got 1 splits when splitting meta index
INFO - Running job: job_201207191341_0007
INFO - map 0% reduce 0%
INFO - map 100% reduce 0%
INFO - map 100% reduce 100%
INFO - Job complete: job_201207191341_0007
INFO - Counters: 26
INFO - Job Counters
INFO - Launched reduce tasks=1
INFO - SLOTS_MILLIS_MAPS=16299
INFO - Total time spent by all reduces waiting after reserving slots (ms)=0
INFO - Total time spent by all maps waiting after reserving slots (ms)=0
INFO - Rack-local map tasks=1
INFO - Launched map tasks=1
INFO - SLOTS_MILLIS_REDUCES=8381
INFO - File Input Format Counters
INFO - Bytes Read=31975
INFO - File Output Format Counters
INFO - Bytes Written=0
INFO - FileSystemCounters
INFO - FILE_BYTES_READ=12006
INFO - HDFS_BYTES_READ=38075
INFO - FILE_BYTES_WRITTEN=95689
INFO - HDFS_BYTES_WRITTEN=24900
INFO - Map-Reduce Framework
INFO - Map output materialized bytes=12006
INFO - Map input records=300
INFO - Reduce shuffle bytes=0
INFO - Spilled Records=600
INFO - Map output bytes=11400
INFO - Map input bytes=0
INFO - Combine input records=0
INFO - SPLIT_RAW_BYTES=181
INFO - Reduce input records=300
INFO - Reduce input groups=300
INFO - Combine output records=0
INFO - Reduce output records=0
INFO - Map output records=300
INFO - Time Taken = 36 seconds
Time Taken = 102 seconds
Time elapsed: 102.624 seconds.
---------------------

Thanks for you assistance and time Craig, you have been very helpful!

- D.D.

Options: ReplyQuote


Sorry, only registered users may post in this forum.
This forum powered by Phorum.