[TR-474] getNumberOfTokens of the class UpdatingCollectionStatistics gives back the number of pointers instead of the number of tokens. Created: 27/Apr/17 Updated: 11/Jul/18 Resolved: 11/Jul/18
|Reporter:||Andrea Langeli||Assignee:||Craig Macdonald|
|Attachments:||Approach_1.java Approach_2.java Approach_3.java Index.java|
There is a bug in the method getNumberOfTokens of the class UpdatingCollectionStatistics within the Index class. This method returns the value associated with the property "num.Pointers" instead of the one related to "num.Tokens". This originates a problem when IndexOnDisk objects are merged to create a new index: its Properties object indicates a number of tokens smaller than the right one, hence the average document length of the collection documents turns out to be wrong.
I upload the patched Java class.
|Comment by Richard McCreadie [ 27/Apr/17 ]|
Good spot on that bug. I can confirm that Index.class is reading the wrong property. Index.class line 90.
return Long.parseLong(properties.getProperty("num.Pointers", "0"));
return Long.parseLong(properties.getProperty("num.Tokens", "0"));
|Comment by Craig Macdonald [ 28/Apr/17 ]|
Thanks both. Andrea, could you tell us how you found the bug (which particular indexing mechanism, as Terrier has a few!). This will enable us to generate a verifiable unit test case.
|Comment by Andrea Langeli [ 29/Apr/17 ]|
Yes, of course. I have created a collection of 4 PDF documents, each one composed of an only statement. I have indexed the collection using three different approaches and finally I have applied the same query to each Index object I have gotten previously. The query consisted of four words: the first one occurred one time in the first document, the second word occurred one time in the third document. The indexing strategies I have used were:
In all three cases I have adopted the tf-idf weighting model. The first two approaches gave back the same results (the same ranking list and the same values), instead the third one returned slightly lower scores than the first two cases. Gianmaria Silvello and I have thought that this mismatch was due to the normalization: we have inspected the code, searching for a solution.
The bug is in the method getNumberOfTokens of the inner class UpdatingCollectionStatistics. I report below the steps of the indexing process before the occurrence of the error.
(IncrementaIndex) indexDocument(Document doc) – (IncrementalIndex) flush() – (IncrementalIndex) flush ((Runnable) mergePolicy).run() – (IncrementalMergeSingle) run merger.mergeStructures() – (StructureMerger) mergeStructures createLexidFile() – (StructureMerger) createLexidFile LexiconBuilder.optimise(destIndex, “lexicon”) – (LexiconBuilder) optimise counter.close() – (BasicLexiconCollectionStatisticsCounter) close.
The method close of the static class BasicLexiconCollectionStatisticsCounter modifies the Properties object associated with the destination IndexOnDisk object setting “num.Documents”, “num.Tokens”, “num.Terms” to the corresponding values of the BasicLexiconCollectionStatisticsCounter variables. This modifies indirectly the values of the variables numberOfDocuments, numberOfTokens, numberOfUniqueTerms of the UpdatingCollectionStatistics object related to the IndexOnDisk object. These values are used for evaluate the score assigned to each collection document during the retrieval process.
I am Andrea Langeli, a computer engineering student of the University of Padua. I’m working on Terrier for my master’s thesis. My supervisor is Gianmaria Silvello.
|Comment by Craig Macdonald [ 11/Jul/18 ]|
This was a dup. Already fixed in