org.terrier.indexing

Class WARC10Collection

java.lang.Object

org.terrier.indexing.MultiDocumentFileCollection

org.terrier.indexing.WARC018Collection

org.terrier.indexing.WARC10Collection

All Implemented Interfaces:

Closeable, AutoCloseable, Collection

public class WARC10Collection
extends WARC018Collection

This object is used to parse WARC format web crawls, version 0.10. Uses properties from WARC018Collection.

Author:

Craig Macdonald

Field Summary

Fields inherited from class org.terrier.indexing.WARC018Collection

currentDocumentBlobLength, warc_crawldate_header, warc_docno_header, warc_url_header

Fields inherited from class org.terrier.indexing.MultiDocumentFileCollection

currentFilename, desiredEncoding, DocProperties, documentClass, documentsInThisFile, eoc, eof, FileNumber, FilesToProcess, forceUTF8, is, logger, SkipFile, tokeniser

Constructor Summary

Constructors

Constructor and Description
WARC10Collection()
WARC10Collection(InputStream input)
WARC10Collection(List<String> files, String TagSet, String BlacklistSpecFilename, String ignored)
WARC10Collection(String CollectionSpecFilename)
WARC10Collection(String CollectionSpecFilename, String TagSet, String BlacklistSpecFilename, String ignored)

Method Summary

Modifier and Type	Method and Description
boolean	nextDocument() Move the collection to the start of the next document.
protected void	processRedirect(String source, String target)

Methods inherited from class org.terrier.indexing.WARC018Collection

getDocid, getDocument, parseHeaders, readLine

Methods inherited from class org.terrier.indexing.MultiDocumentFileCollection

close, endOfCollection, extractCharset, hasNext, loadDocumentClass, next, openNewFile, openNextFile, reset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.terrier.indexing.Collection

endOfCollection, reset

Methods inherited from interface java.io.Closeable

close

Constructor Detail

WARC10Collection

public WARC10Collection()

WARC10Collection

public WARC10Collection(InputStream input)

WARC10Collection

public WARC10Collection(String CollectionSpecFilename)

WARC10Collection

public WARC10Collection(List<String> files,
                        String TagSet,
                        String BlacklistSpecFilename,
                        String ignored)

WARC10Collection

public WARC10Collection(String CollectionSpecFilename,
                        String TagSet,
                        String BlacklistSpecFilename,
                        String ignored)

Method Detail

processRedirect

protected void processRedirect(String source,
                               String target)

nextDocument

public boolean nextDocument()

Move the collection to the start of the next document.

Specified by:

nextDocument in interface Collection

Overrides:

nextDocument in class WARC018Collection

Returns:

boolean true if there exists another document in the collection, otherwise it returns false.