org.terrier.indexing

Class WARC018Collection

java.lang.Object

org.terrier.indexing.MultiDocumentFileCollection

org.terrier.indexing.WARC018Collection

All Implemented Interfaces:

Closeable, AutoCloseable, Collection

Direct Known Subclasses:

WARC10Collection

public class WARC018Collection
extends MultiDocumentFileCollection
implements Collection

This object is used to parse WARC format web crawls, 0.18. The precise Document class to be used can be specified with the trec.document.class property.

Properties

• trec.document.class the Document class to parse individual documents (defaults to TaggedDocument).
• warc018collection.force.utf8 - should UTF8 encoding be assumed throughout. Defaults to false.
• warc018collection.header.docno - what header has the thing to be used as docno? Defaults to warc-trec-id.
• warc018collection.header.url - what header has the thing to be used as url? Defaults to warc-target-url.

Author:

Craig Macdonald

Field Summary

Fields

Modifier and Type	Field and Description
protected long	currentDocumentBlobLength the length of the blob containing the document data
protected String	warc_crawldate_header what header for the crawldate document metadata
protected String	warc_docno_header what header for the docno document metadata
protected String	warc_url_header what header for the url document metadata

Fields inherited from class org.terrier.indexing.MultiDocumentFileCollection

currentFilename, desiredEncoding, DocProperties, documentClass, documentsInThisFile, eoc, eof, FileNumber, FilesToProcess, forceUTF8, is, logger, SkipFile, tokeniser

Constructor Summary

Constructors

Constructor and Description
WARC018Collection()
WARC018Collection(InputStream input)
WARC018Collection(List<String> files)
WARC018Collection(List<String> files, String TagSet, String BlacklistSpecFilename, String ignored)
WARC018Collection(String CollectionSpecFilename)
WARC018Collection(String CollectionSpecFilename, String TagSet, String BlacklistSpecFilename, String ignored)

Method Summary

Modifier and Type	Method and Description
String	getDocid() Get the String document identifier of the current document.
Document	getDocument() Get the document object representing the current document.
boolean	nextDocument() Move the collection to the start of the next document.
protected int	parseHeaders(boolean requireContentLength)
protected String	readLine() read a line from the currently open InputStream is

Methods inherited from class org.terrier.indexing.MultiDocumentFileCollection

close, endOfCollection, extractCharset, hasNext, loadDocumentClass, next, openNewFile, openNextFile, reset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.terrier.indexing.Collection

endOfCollection, reset

Methods inherited from interface java.io.Closeable

close

Field Detail

currentDocumentBlobLength

protected long currentDocumentBlobLength

the length of the blob containing the document data

warc_docno_header

protected final String warc_docno_header

what header for the docno document metadata

warc_url_header

protected final String warc_url_header

what header for the url document metadata

warc_crawldate_header

protected final String warc_crawldate_header

what header for the crawldate document metadata

Constructor Detail

WARC018Collection

public WARC018Collection()

WARC018Collection

public WARC018Collection(InputStream input)

WARC018Collection

public WARC018Collection(String CollectionSpecFilename)

WARC018Collection

public WARC018Collection(List<String> files)

WARC018Collection

public WARC018Collection(List<String> files,
                         String TagSet,
                         String BlacklistSpecFilename,
                         String ignored)

WARC018Collection

public WARC018Collection(String CollectionSpecFilename,
                         String TagSet,
                         String BlacklistSpecFilename,
                         String ignored)

Method Detail

nextDocument

public boolean nextDocument()

Move the collection to the start of the next document.

Specified by:

nextDocument in interface Collection

Specified by:

nextDocument in class MultiDocumentFileCollection

Returns:

boolean true if there exists another document in the collection, otherwise it returns false.

getDocid

public String getDocid()

Get the String document identifier of the current document.

getDocument

public Document getDocument()

Get the document object representing the current document.

Specified by:

getDocument in interface Collection

Specified by:

getDocument in class MultiDocumentFileCollection

Returns:

Document the current document;

parseHeaders

protected int parseHeaders(boolean requireContentLength)
                    throws IOException

Throws:

IOException

readLine

protected String readLine()
                   throws IOException

read a line from the currently open InputStream is

Throws:

IOException