Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-137

TRECCollection cannot add properties from the document tags to the meta index at indexing time

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0
    • Fix Version/s: 3.5
    • Component/s: .indexing
    • Labels:
      None

      Description

      Modified the TRECCollection class such that additional document tags like the URL can be saved automatically into the meta index. Initial patch provided.

      Example: I want to index the title field and add the url to the meta index for the collection below:
      <doc>
      <docno>NYTMay2006-0</docno>
      <url>http://events.nytimes.com/2006/05/03/dining/reviews/03rest.html&lt;/url>
      <title>Forum: Dine Out</title>
      </doc>

      Set trec.properties:
      # Which fields do we want to pass to the indexer as properties (in addition to the docno)?
      TrecDocTags.propertytags=url
      # At the indexer which keys to we add to the Meta index?
      indexer.meta.forward.keys=docno,url
      # What are the lengths of those keys (e.g. the url will be much longer)?
      indexer.meta.forward.keylens=20,100

        Attachments

          Activity

          richardm Richard McCreadie created issue -
          richardm Richard McCreadie made changes -
          Field Original Value New Value
          Assignee Iadh Ounis [ ounis ] Richard McCreadie [ richardm ]
          richardm Richard McCreadie made changes -
          Description Modified the TRECCollection class such that additional document tags like the URL can be saved automatically into the meta index. Initial patch provided.

          Example: I want to index the title field and add the url to the meta index for the collection below:
          <doc>
          <docno>NYTMay2006-0</docno>
          <url>http://events.nytimes.com/2006/05/03/dining/reviews/03rest.html&lt;/url>
          <title>Forum: Dine Out</title>
          </doc>

          Set trec.properties:
          # Which fields do we want to pass to the indexer as properties (in addition to the docno)?
          TrecDocTags.propertytags=url
          # At the indexer which keys to we add to the Meta index?
          indexer.meta.forward.keys=docno,desc
          # What are the lengths of those keys (e.g. the url will be much longer)?
          indexer.meta.forward.keylens=20,100
          Modified the TRECCollection class such that additional document tags like the URL can be saved automatically into the meta index. Initial patch provided.

          Example: I want to index the title field and add the url to the meta index for the collection below:
          <doc>
          <docno>NYTMay2006-0</docno>
          <url>http://events.nytimes.com/2006/05/03/dining/reviews/03rest.html&lt;/url>
          <title>Forum: Dine Out</title>
          </doc>

          Set trec.properties:
          # Which fields do we want to pass to the indexer as properties (in addition to the docno)?
          TrecDocTags.propertytags=url
          # At the indexer which keys to we add to the Meta index?
          indexer.meta.forward.keys=docno,url
          # What are the lengths of those keys (e.g. the url will be much longer)?
          indexer.meta.forward.keylens=20,100
          Hide
          craigm Craig Macdonald added a comment -

          Tagged for 3.1

          Show
          craigm Craig Macdonald added a comment - Tagged for 3.1
          craigm Craig Macdonald made changes -
          Affects Version/s 3.1 [ 10021 ]
          Affects Version/s 3.0 [ 10020 ]
          Hide
          craigm Craig Macdonald added a comment -

          I reviewed this. My comments are as follows:

          1. Much of the property tag calculations occurs for each document parsed. This is very expensive. Instead, some arrays could be populated in setTags() :

          protected int[] propertyTagLengths;
          protected char[][] startPropertyTags;
          protected char[][] endPropertyTags;	
          

          2. Additionally, could a method be used that did the same for the DOCNO and each of the property tags.

          3. Can you give a look at ensuring all of the tag-related are adequately described within the javadoc for TRECCollection.

          Ta.

          Show
          craigm Craig Macdonald added a comment - I reviewed this. My comments are as follows: 1. Much of the property tag calculations occurs for each document parsed. This is very expensive. Instead, some arrays could be populated in setTags() : protected int [] propertyTagLengths; protected char [][] startPropertyTags; protected char [][] endPropertyTags; 2. Additionally, could a method be used that did the same for the DOCNO and each of the property tags. 3. Can you give a look at ensuring all of the tag-related are adequately described within the javadoc for TRECCollection. Ta.
          craigm Craig Macdonald made changes -
          Affects Version/s 3.0 [ 10020 ]
          Affects Version/s 3.1 [ 10021 ]
          Fix Version/s 3.1 [ 10021 ]
          Fix Version/s 3.0 [ 10020 ]
          Hide
          richardm Richard McCreadie added a comment -

          New patch. Moved tag computation into setTags(). Refactored getDocument to move the tag scan code into a separate getTag() method. Passes the testSingleDocumentSingleTermProperyTags() test case.

          Show
          richardm Richard McCreadie added a comment - New patch. Moved tag computation into setTags(). Refactored getDocument to move the tag scan code into a separate getTag() method. Passes the testSingleDocumentSingleTermProperyTags() test case.
          richardm Richard McCreadie made changes -
          Attachment TREC-178 v2.patch [ 10261 ]
          Hide
          craigm Craig Macdonald added a comment -

          Ok. Looks very good. Three minor comments:
          1. Can you add documentation to TRECCollection, and probably configure_indexing.html
          2. TrecDocTags should not be hard-coded
          3. Can you include the updated test case in the patch?

          Show
          craigm Craig Macdonald added a comment - Ok. Looks very good. Three minor comments: 1. Can you add documentation to TRECCollection, and probably configure_indexing.html 2. TrecDocTags should not be hard-coded 3. Can you include the updated test case in the patch?
          Hide
          richardm Richard McCreadie added a comment -

          Added JavsDoc to TRECCollection, fixed hard codeing of TRECDocTags and updated a new test case

          Show
          richardm Richard McCreadie added a comment - Added JavsDoc to TRECCollection, fixed hard codeing of TRECDocTags and updated a new test case
          richardm Richard McCreadie made changes -
          Attachment TREC-178v3.patch [ 10262 ]
          Hide
          craigm Craig Macdonald added a comment -

          Committed. Thanks for your patience Richard!

          Show
          craigm Craig Macdonald added a comment - Committed. Thanks for your patience Richard!
          craigm Craig Macdonald made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          craigm Craig Macdonald made changes -
          Project TREC [ 10010 ] Terrier Core [ 10000 ]
          Key TREC-178 TR-137
          Workflow jira [ 10426 ] Terrier Open Source [ 10530 ]
          Affects Version/s 3.0 [ 10030 ]
          Affects Version/s 3.0 [ 10020 ]
          Component/s .indexing [ 10002 ]
          Component/s Core [ 10020 ]
          Fix Version/s 3.1 [ 10040 ]
          Fix Version/s 3.1 [ 10021 ]

            People

            • Assignee:
              richardm Richard McCreadie
              Reporter:
              richardm Richard McCreadie
            • Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: