Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-137

TRECCollection cannot add properties from the document tags to the meta index at indexing time

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0
    • Fix Version/s: 3.5
    • Component/s: .indexing
    • Labels:
      None

      Description

      Modified the TRECCollection class such that additional document tags like the URL can be saved automatically into the meta index. Initial patch provided.

      Example: I want to index the title field and add the url to the meta index for the collection below:
      <doc>
      <docno>NYTMay2006-0</docno>
      <url>http://events.nytimes.com/2006/05/03/dining/reviews/03rest.html&lt;/url>
      <title>Forum: Dine Out</title>
      </doc>

      Set trec.properties:
      # Which fields do we want to pass to the indexer as properties (in addition to the docno)?
      TrecDocTags.propertytags=url
      # At the indexer which keys to we add to the Meta index?
      indexer.meta.forward.keys=docno,url
      # What are the lengths of those keys (e.g. the url will be much longer)?
      indexer.meta.forward.keylens=20,100

        Attachments

        1. TREC-178 v2.patch
          8 kB
          Richard McCreadie
        2. TREC-178v3.patch
          11 kB
          Richard McCreadie
        3. TRECMetaProperties.patch
          4 kB
          Richard McCreadie

          Activity

          Hide
          craigm Craig Macdonald added a comment -

          Tagged for 3.1

          Show
          craigm Craig Macdonald added a comment - Tagged for 3.1
          Hide
          craigm Craig Macdonald added a comment -

          I reviewed this. My comments are as follows:

          1. Much of the property tag calculations occurs for each document parsed. This is very expensive. Instead, some arrays could be populated in setTags() :

          protected int[] propertyTagLengths;
          protected char[][] startPropertyTags;
          protected char[][] endPropertyTags;	
          

          2. Additionally, could a method be used that did the same for the DOCNO and each of the property tags.

          3. Can you give a look at ensuring all of the tag-related are adequately described within the javadoc for TRECCollection.

          Ta.

          Show
          craigm Craig Macdonald added a comment - I reviewed this. My comments are as follows: 1. Much of the property tag calculations occurs for each document parsed. This is very expensive. Instead, some arrays could be populated in setTags() : protected int [] propertyTagLengths; protected char [][] startPropertyTags; protected char [][] endPropertyTags; 2. Additionally, could a method be used that did the same for the DOCNO and each of the property tags. 3. Can you give a look at ensuring all of the tag-related are adequately described within the javadoc for TRECCollection. Ta.
          Hide
          richardm Richard McCreadie added a comment -

          New patch. Moved tag computation into setTags(). Refactored getDocument to move the tag scan code into a separate getTag() method. Passes the testSingleDocumentSingleTermProperyTags() test case.

          Show
          richardm Richard McCreadie added a comment - New patch. Moved tag computation into setTags(). Refactored getDocument to move the tag scan code into a separate getTag() method. Passes the testSingleDocumentSingleTermProperyTags() test case.
          Hide
          craigm Craig Macdonald added a comment -

          Ok. Looks very good. Three minor comments:
          1. Can you add documentation to TRECCollection, and probably configure_indexing.html
          2. TrecDocTags should not be hard-coded
          3. Can you include the updated test case in the patch?

          Show
          craigm Craig Macdonald added a comment - Ok. Looks very good. Three minor comments: 1. Can you add documentation to TRECCollection, and probably configure_indexing.html 2. TrecDocTags should not be hard-coded 3. Can you include the updated test case in the patch?
          Hide
          richardm Richard McCreadie added a comment -

          Added JavsDoc to TRECCollection, fixed hard codeing of TRECDocTags and updated a new test case

          Show
          richardm Richard McCreadie added a comment - Added JavsDoc to TRECCollection, fixed hard codeing of TRECDocTags and updated a new test case
          Hide
          craigm Craig Macdonald added a comment -

          Committed. Thanks for your patience Richard!

          Show
          craigm Craig Macdonald added a comment - Committed. Thanks for your patience Richard!

            People

            • Assignee:
              richardm Richard McCreadie
              Reporter:
              richardm Richard McCreadie
            • Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: