WARC018Collection (Terrier 4.0 API)

java.lang.Object
- org.terrier.indexing.WARC018Collection

All Implemented Interfaces:

Closeable, AutoCloseable, Collection

Direct Known Subclasses:

WARC10Collection
```
public class WARC018Collection
extends Object
implements Collection
```
This object is used to parse WARC format web crawls, 0.18. The precise Document class to be used can be specified with the trec.document.class property.
Properties
- trec.document.class the Document class to parse individual documents (defaults to TaggedDocument).
- warc018collection.force.utf8 - should UTF8 encoding be assumed throughout. Defaults to false.
- warc018collection.header.docno - what header has the thing to be used as docno? Defaults to warc-trec-id.
- warc018collection.header.url - what header has the thing to be used as url? Defaults to warc-target-url.
Author:

Craig Macdonald

Field Summary

Fields
Modifier and Type	Field and Description
`protected long`	`currentDocumentBlobLength` the length of the blob containing the document data
`protected String`	`desiredEncoding` Encoding to be used to open all files.
`protected Map<String,String>`	`DocProperties` properties for the current document
`protected Class<? extends Document>`	`documentClass` Class to use for all documents parsed by this class
`protected int`	`documentsInThisFile` Counts the number of documents that have been found in this file.
`protected boolean`	`eoc` are we at the end of the collection?
`protected boolean`	`eof` has the end of the current input file been reached?
`protected int`	`FileNumber` The index in the FilesToProcess of the currently processed file.
`protected ArrayList<String>`	`FilesToProcess` The list of files to process.
`protected boolean`	`forceUTF8` should UTF8 encoding be assumed?
`protected InputStream`	`is` the input stream of the current input file
`protected static org.apache.log4j.Logger`	`logger` logger for this class
`protected Tokeniser`	`tokeniser` Tokeniser to use for all documents parsed by this class
`protected String`	`warc_crawldate_header` what header for the crawldate document metadata
`protected String`	`warc_docno_header` what header for the docno document metadata
`protected String`	`warc_url_header` what header for the url document metadata

Constructor Summary

Constructors
Constructor and Description
`WARC018Collection()` default constructor for this collection object.
`WARC018Collection(InputStream input)` A constructor that reads only the specificed InputStream.
`WARC018Collection(String CollectionSpecFilename)` construct a collection from the denoted collection.spec file

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`close()` Closes the collection, any files that may be open.
`boolean`	`endOfCollection()` Returns true if the end of the collection has been reached
`String`	`getDocid()` Get the String document identifier of the current document.
`Document`	`getDocument()` Get the document object representing the current document.
`boolean`	`hasNext()` Check whether it is the last document in the collection
`protected void`	`loadDocumentClass()` Loads the class that will supply all documents for this Collection.
`Document`	`next()` Return the next document
`boolean`	`nextDocument()` Move the collection to the start of the next document.
`protected boolean`	`openNextFile()` Opens the next document from the collection specification.
`protected int`	`parseHeaders(boolean requireContentLength)`
`protected void`	`readCollectionSpec(String CollectionSpecFilename)` read in the collection.spec
`protected String`	`readLine()` read a line from the currently open InputStream is
`void`	`reset()` Resets the Collection iterator to the start of the collection.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - logger
```
protected static final org.apache.log4j.Logger logger
```
    logger for this class
  - documentsInThisFile
```
protected int documentsInThisFile
```
    Counts the number of documents that have been found in this file.
  - eoc
```
protected boolean eoc
```
    are we at the end of the collection?
  - eof
```
protected boolean eof
```
    has the end of the current input file been reached?
  - is
```
protected InputStream is
```
    the input stream of the current input file
  - currentDocumentBlobLength
```
protected long currentDocumentBlobLength
```
    the length of the blob containing the document data
  - DocProperties
```
protected Map<String,String> DocProperties
```
    properties for the current document
  - FilesToProcess
```
protected ArrayList<String> FilesToProcess
```
    The list of files to process.
  - FileNumber
```
protected int FileNumber
```
    The index in the FilesToProcess of the currently processed file.
  - forceUTF8
```
protected final boolean forceUTF8
```
    should UTF8 encoding be assumed?
  - warc_docno_header
```
protected final String warc_docno_header
```
    what header for the docno document metadata
  - warc_url_header
```
protected final String warc_url_header
```
    what header for the url document metadata
  - warc_crawldate_header
```
protected final String warc_crawldate_header
```
    what header for the crawldate document metadata
  - desiredEncoding
```
protected String desiredEncoding
```
    Encoding to be used to open all files.
  - documentClass
```
protected Class<? extends Document> documentClass
```
    Class to use for all documents parsed by this class
  - tokeniser
```
protected Tokeniser tokeniser
```
    Tokeniser to use for all documents parsed by this class
- Constructor Detail
  - WARC018Collection
```
public WARC018Collection()
```
    default constructor for this collection object. Reads files from the system default collection.spec file
  - WARC018Collection
```
public WARC018Collection(String CollectionSpecFilename)
```
    construct a collection from the denoted collection.spec file
  - WARC018Collection
```
public WARC018Collection(InputStream input)
```
    A constructor that reads only the specificed InputStream.
- Method Detail
  - hasNext
```
public boolean hasNext()
```
    Check whether it is the last document in the collection
    
    Returns:
    boolean
  - next
```
public Document next()
```
    Return the next document
    
    Returns:
    next document
  - close
```
public void close()
```
    Closes the collection, any files that may be open.
    
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
  - endOfCollection
```
public boolean endOfCollection()
```
    Returns true if the end of the collection has been reached
    
    Specified by:
    
    endOfCollection in interface Collection
    
    Returns:
    boolean true if the end of collection has been reached, otherwise it returns false.
  - getDocid
```
public String getDocid()
```
    Get the String document identifier of the current document.
  - loadDocumentClass
```
protected void loadDocumentClass()
```
    Loads the class that will supply all documents for this Collection. Set by property trec.document.class
  - getDocument
```
public Document getDocument()
```
    Get the document object representing the current document.
    
    Specified by:
    
    getDocument in interface Collection
    
    Returns:
    Document the current document;
  - parseHeaders
```
protected int parseHeaders(boolean requireContentLength)
                    throws IOException
```
    Throws:
    
    IOException
  - nextDocument
```
public boolean nextDocument()
```
    Move the collection to the start of the next document.
    
    Specified by:
    
    nextDocument in interface Collection
    
    Returns:
    boolean true if there exists another document in the collection, otherwise it returns false.
  - readLine
```
protected String readLine()
                   throws IOException
```
    read a line from the currently open InputStream is
    
    Throws:
    
    IOException
  - openNextFile
```
protected boolean openNextFile()
                        throws IOException
```
    Opens the next document from the collection specification.
    
    Returns:
    boolean true if the file was opened successufully. If there are no more files to open, it returns false.
    
    Throws:
    
    IOException - if there is an exception while opening the collection files.
  - readCollectionSpec
```
protected void readCollectionSpec(String CollectionSpecFilename)
```
    read in the collection.spec
  - reset
```
public void reset()
```
    Resets the Collection iterator to the start of the collection.
    
    Specified by:
    
    reset in interface Collection

Class WARC018Collection

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

logger

documentsInThisFile

eoc

eof

is

currentDocumentBlobLength

DocProperties

FilesToProcess

FileNumber

forceUTF8

warc_docno_header

warc_url_header

warc_crawldate_header

desiredEncoding

documentClass

tokeniser

Constructor Detail

WARC018Collection

WARC018Collection

WARC018Collection

Method Detail

hasNext

next

close

endOfCollection

getDocid

loadDocumentClass

getDocument

parseHeaders

nextDocument

readLine

openNextFile

readCollectionSpec

reset