Terrier Users :  Terrier Forum terrier.org
General discussion about using/developing applications using Terrier 
how to use terrier4.1 for indexing large document? how to set properties of terrier.
Posted by: ll1042668699 ()
Date: December 06, 2016 11:07AM

Hi friends:
I encounter this problem,and i have no idea to fix it,can you help me? thanks.
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at gnu.trove.TObjectIntHashMap.rehash(TObjectIntHashMap.java:172)
at gnu.trove.THash.postInsertHook(THash.java:359)
at gnu.trove.TObjectIntHashMap.adjustOrPutValue(TObjectIntHashMap.java:537)
at org.terrier.structures.indexing.LexiconMap$1.execute(LexiconMap.java:89)
at org.terrier.structures.indexing.LexiconMap$1.execute(LexiconMap.java:85)
at gnu.trove.TObjectIntHashMap.forEachEntry(TObjectIntHashMap.java:426)
at org.terrier.structures.indexing.DocumentPostingList.forEachTerm(DocumentPostingList.java:131)
at org.terrier.structures.indexing.LexiconMap.insert(LexiconMap.java:85)
at org.terrier.structures.indexing.LexiconBuilder.addDocumentTerms(LexiconBuilder.java:364)
at org.terrier.structures.indexing.classical.BasicIndexer.indexDocument(BasicIndexer.java:386)
at org.terrier.structures.indexing.classical.BasicIndexer.createDirectIndex(BasicIndexer.java:277)
at org.terrier.structures.indexing.Indexer.index(Indexer.java:348)
at org.terrier.applications.TRECIndexing.index(TRECIndexing.java:174)
at org.terrier.applications.TrecTerrier.run(TrecTerrier.java:408)
at org.terrier.applications.TrecTerrier.applyOptions(TrecTerrier.java:599)
at org.terrier.applications.TrecTerrier.main(TrecTerrier.java:246)

Options: ReplyQuote
Re: how to use terrier4.1 for indexing large document? how to set properties of terrier.
Posted by: lupacexi ()
Date: August 15, 2017 03:46PM

Do anyone known?

Options: ReplyQuote
Re: how to use terrier4.1 for indexing large document? how to set properties of terrier.
Posted by: craigm ()
Date: August 16, 2017 11:23AM

You need to use more memory during indexing. E.g. set the TERRIER_HEAP_MEM=8G env variable.

Craig

Options: ReplyQuote


Sorry, only registered users may post in this forum.
This forum powered by Phorum.