WARC09Collection (Terrier 4.0 API)

java.lang.Object
- org.terrier.indexing.WARC09Collection

All Implemented Interfaces:

Closeable, AutoCloseable, Collection
```
public class WARC09Collection
extends Object
implements Collection
```
This object is used to parse WARC format web crawls, version 0.9. The precise Document class to be used can be specified with the trec.document.class property. The following links denote the pages that were used to construct the format of this object: http://www.yr-bcn.es/webspam/datasets/uk2006-pages/excerpt.txt http://archive-access.sourceforge.net/warc/warc_file_format.html http://crawler.archive.org/apidocs/index.html?org/archive/io/arc/ARCWriter.html http://crawler.archive.org/apidocs/org/archive/io/GzippedInputStream.html
Properties
- trec.document.class the Document class to parse individual documents (defaults to TaggedDocument).
Author:

Craig Macdonald

Field Summary

Fields
Modifier and Type	Field and Description
`protected String`	`currentDocno` the document number of the current document
`protected long`	`currentDocumentBlobLength` the length of the blob containing the document data
`protected Map<String,String>`	`DocProperties` properties for the current document
`protected Class<? extends Document>`	`documentClass` Class to use for all documents parsed by this class
`protected int`	`documentsInThisFile` Counts the number of documents that have been found in this file.
`protected boolean`	`eoc` are we at the end of the collection?
`protected boolean`	`eof` has the end of the current input file been reached?
`protected int`	`FileNumber` The index in the FilesToProcess of the currently processed file.
`protected ArrayList<String>`	`FilesToProcess` The list of files to process.
`protected InputStream`	`is` the input stream of the current input file
`protected static org.apache.log4j.Logger`	`logger` logger for this class
`protected Tokeniser`	`tokeniser` Tokeniser to use for all documents parsed by this class

Constructor Summary

Constructors
Constructor and Description
`WARC09Collection()` default constructor for this collection object.
`WARC09Collection(InputStream input)` A constructor that reads only the specificed InputStream.
`WARC09Collection(String CollectionSpecFilename)` construct a collection from the denoted collection.spec file

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`close()` Closes the collection, any files that may be open.
`boolean`	`endOfCollection()` Returns true if the end of the collection has been reached
`String`	`getDocid()` Get the String document identifier of the current document.
`Document`	`getDocument()` Get the document object representing the current document.
`boolean`	`hasNext()` Check whether it is the last document in the collection
`protected void`	`loadDocumentClass()` Loads the class that will supply all documents for this Collection.
`Document`	`next()` Return the next document
`boolean`	`nextDocument()` Move the collection to the start of the next document.
`protected boolean`	`openNextFile()` Opens the next document from the collection specification.
`protected void`	`readCollectionSpec(String CollectionSpecFilename)` read in the collection.spec
`protected String`	`readLine()` read a line from the currently open InputStream is
`void`	`reset()` Resets the Collection iterator to the start of the collection.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - logger
```
protected static final org.apache.log4j.Logger logger
```
    logger for this class
  - documentsInThisFile
```
protected int documentsInThisFile
```
    Counts the number of documents that have been found in this file.
  - eoc
```
protected boolean eoc
```
    are we at the end of the collection?
  - eof
```
protected boolean eof
```
    has the end of the current input file been reached?
  - currentDocno
```
protected String currentDocno
```
    the document number of the current document
  - is
```
protected InputStream is
```
    the input stream of the current input file
  - currentDocumentBlobLength
```
protected long currentDocumentBlobLength
```
    the length of the blob containing the document data
  - DocProperties
```
protected Map<String,String> DocProperties
```
    properties for the current document
  - FilesToProcess
```
protected ArrayList<String> FilesToProcess
```
    The list of files to process.
  - FileNumber
```
protected int FileNumber
```
    The index in the FilesToProcess of the currently processed file.
  - documentClass
```
protected Class<? extends Document> documentClass
```
    Class to use for all documents parsed by this class
  - tokeniser
```
protected Tokeniser tokeniser
```
    Tokeniser to use for all documents parsed by this class
- Constructor Detail
  - WARC09Collection
```
public WARC09Collection()
```
    default constructor for this collection object. Reads files from the system default collection.spec file
  - WARC09Collection
```
public WARC09Collection(InputStream input)
```
    A constructor that reads only the specificed InputStream.
  - WARC09Collection
```
public WARC09Collection(String CollectionSpecFilename)
```
    construct a collection from the denoted collection.spec file
- Method Detail
  - loadDocumentClass
```
protected void loadDocumentClass()
```
    Loads the class that will supply all documents for this Collection. Set by property trec.document.class
  - hasNext
```
public boolean hasNext()
```
    Check whether it is the last document in the collection
    
    Returns:
    boolean
  - next
```
public Document next()
```
    Return the next document
    
    Returns:
    next document
  - close
```
public void close()
```
    Closes the collection, any files that may be open.
    
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
  - endOfCollection
```
public boolean endOfCollection()
```
    Returns true if the end of the collection has been reached
    
    Specified by:
    
    endOfCollection in interface Collection
    
    Returns:
    boolean true if the end of collection has been reached, otherwise it returns false.
  - getDocid
```
public String getDocid()
```
    Get the String document identifier of the current document.
  - getDocument
```
public Document getDocument()
```
    Get the document object representing the current document.
    
    Specified by:
    
    getDocument in interface Collection
    
    Returns:
    Document the current document;
  - nextDocument
```
public boolean nextDocument()
```
    Move the collection to the start of the next document.
    
    Specified by:
    
    nextDocument in interface Collection
    
    Returns:
    boolean true if there exists another document in the collection, otherwise it returns false.
  - readLine
```
protected String readLine()
                   throws IOException
```
    read a line from the currently open InputStream is
    
    Throws:
    
    IOException
  - openNextFile
```
protected boolean openNextFile()
                        throws IOException
```
    Opens the next document from the collection specification.
    
    Returns:
    boolean true if the file was opened successufully. If there are no more files to open, it returns false.
    
    Throws:
    
    IOException - if there is an exception while opening the collection files.
  - readCollectionSpec
```
protected void readCollectionSpec(String CollectionSpecFilename)
```
    read in the collection.spec
  - reset
```
public void reset()
```
    Resets the Collection iterator to the start of the collection.
    
    Specified by:
    
    reset in interface Collection

Class WARC09Collection

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

logger

documentsInThisFile

eoc

eof

currentDocno

is

currentDocumentBlobLength

DocProperties

FilesToProcess

FileNumber

documentClass

tokeniser

Constructor Detail

WARC09Collection

WARC09Collection

WARC09Collection

Method Detail

loadDocumentClass

hasNext

next

close

endOfCollection

getDocid

getDocument

nextDocument

readLine

openNextFile

readCollectionSpec

reset