org.terrier.indexing

Class WARC09Collection

java.lang.Object

org.terrier.indexing.MultiDocumentFileCollection

org.terrier.indexing.WARC09Collection

All Implemented Interfaces:

Closeable, AutoCloseable, Collection

public class WARC09Collection
extends MultiDocumentFileCollection

This object is used to parse WARC format web crawls, version 0.9. The precise Document class to be used can be specified with the trec.document.class property. The following links denote the pages that were used to construct the format of this object: http://www.yr-bcn.es/webspam/datasets/uk2006-pages/excerpt.txt http://archive-access.sourceforge.net/warc/warc_file_format.html http://crawler.archive.org/apidocs/index.html?org/archive/io/arc/ARCWriter.html http://crawler.archive.org/apidocs/org/archive/io/GzippedInputStream.html

Properties

• trec.document.class the Document class to parse individual documents (defaults to TaggedDocument).

Author:

Craig Macdonald

Field Summary

Fields

Modifier and Type	Field and Description
protected String	currentDocno properties for the current document
protected long	currentDocumentBlobLength

Fields inherited from class org.terrier.indexing.MultiDocumentFileCollection

currentFilename, desiredEncoding, DocProperties, documentClass, documentsInThisFile, eoc, eof, FileNumber, FilesToProcess, forceUTF8, is, logger, SkipFile, tokeniser

Constructor Summary

Constructors

Constructor and Description
WARC09Collection()
WARC09Collection(InputStream input)
WARC09Collection(List<String> files, String TagSet, String BlacklistSpecFilename, String ignored)
WARC09Collection(String CollectionSpecFilename)
WARC09Collection(String CollectionSpecFilename, String TagSet, String BlacklistSpecFilename, String ignored)

Method Summary

Modifier and Type	Method and Description
Document	getDocument() Get the document object representing the current document.
boolean	nextDocument() Move the collection to the start of the next document.
protected String	readLine() read a line from the currently open InputStream is
void	reset() Resets the Collection iterator to the start of the collection.

Methods inherited from class org.terrier.indexing.MultiDocumentFileCollection

close, endOfCollection, extractCharset, hasNext, loadDocumentClass, next, openNewFile, openNextFile

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

currentDocumentBlobLength

protected long currentDocumentBlobLength

currentDocno

protected String currentDocno

properties for the current document

Constructor Detail

WARC09Collection

public WARC09Collection()

WARC09Collection

public WARC09Collection(InputStream input)

WARC09Collection

public WARC09Collection(String CollectionSpecFilename)

WARC09Collection

public WARC09Collection(List<String> files,
                        String TagSet,
                        String BlacklistSpecFilename,
                        String ignored)

WARC09Collection

public WARC09Collection(String CollectionSpecFilename,
                        String TagSet,
                        String BlacklistSpecFilename,
                        String ignored)

Method Detail

getDocument

public Document getDocument()

Get the document object representing the current document.

Specified by:

getDocument in interface Collection

Specified by:

getDocument in class MultiDocumentFileCollection

Returns:

Document the current document;

nextDocument

public boolean nextDocument()

Move the collection to the start of the next document.

Specified by:

nextDocument in interface Collection

Specified by:

nextDocument in class MultiDocumentFileCollection

Returns:

boolean true if there exists another document in the collection, otherwise it returns false.

readLine

protected String readLine()
                   throws IOException

read a line from the currently open InputStream is

Throws:

IOException

reset

public void reset()

Resets the Collection iterator to the start of the collection.

Specified by:

reset in interface Collection

Overrides:

reset in class MultiDocumentFileCollection