Uploaded image for project: 'Terrier Core'
  1. Terrier Core
  2. TR-295

WARC10Collection incorrectly misses some documents

          Details

          • Type: Bug
          • Status: Resolved
          • Priority: Major
          • Resolution: Fixed
          • Affects Version/s: 3.6
          • Fix Version/s: 4.0
          • Component/s: None
          • Labels:
            None

            Description

            ClueWeb12_00/0000tw/0000tw-00.warc.gz should have 24644 records (of which 3 are redirects). Currently WARC10Collection only finds 23377 documents in that file.

              Attachments

                Activity

                Ascending order - Click to sort in descending order
                Hide
                Permalink
                craigm Craig Macdonald added a comment -

                The issue appears to be that line ending characters in WARC records are with DOS line endings (\r\n). We were counting only one byte.

                Show
                craigm Craig Macdonald added a comment - The issue appears to be that line ending characters in WARC records are with DOS line endings (\r\n). We were counting only one byte.
                Hide
                Permalink
                craigm Craig Macdonald added a comment -

                Other problem was that the HTTP status line (/usually/ 14 bytes) was not accounted for in the blob length

                Show
                craigm Craig Macdonald added a comment - Other problem was that the HTTP status line (/usually/ 14 bytes) was not accounted for in the blob length
                Hide
                Permalink
                craigm Craig Macdonald added a comment -

                These files address the issue.

                Show
                craigm Craig Macdonald added a comment - These files address the issue.
                Hide
                Permalink
                craigm Craig Macdonald added a comment -

                Committed r3816

                Show
                craigm Craig Macdonald added a comment - Committed r3816

                  People

                  • Assignee:
                    craigm Craig Macdonald
                    Reporter:
                    craigm Craig Macdonald
                  • Watchers:
                    0 Start watching this issue

                    Dates

                    • Created:
                      Updated:
                      Resolved: