org.terrier.indexing

Class MultiDocumentFileCollection

java.lang.Object

org.terrier.indexing.MultiDocumentFileCollection

All Implemented Interfaces:

Closeable, AutoCloseable, Collection

Direct Known Subclasses:

TRECCollection, WARC018Collection, WARC09Collection

public abstract class MultiDocumentFileCollection
extends Object
implements Collection

Field Summary

Fields

Modifier and Type	Field and Description
protected String	currentFilename Filename of current file
protected String	desiredEncoding Encoding to be used to open all files.
protected Map<String,String>	DocProperties properties for the current document
protected Class<? extends Document>	documentClass Class to use for all documents parsed by this class
protected int	documentsInThisFile Counts the number of documents that have been found in this file.
protected boolean	eoc are we at the end of the collection?
protected boolean	eof has the end of the current input file been reached?
protected int	FileNumber The index in the FilesToProcess of the currently processed file.
protected ArrayList<String>	FilesToProcess The list of files to process.
protected boolean	forceUTF8 should UTF8 encoding be assumed?
protected InputStream	is the input stream of the current input file
protected static org.slf4j.Logger	logger logger for this class
protected boolean	SkipFile A boolean which is true when a new file is open.
protected Tokeniser	tokeniser Tokeniser to use for all documents parsed by this class

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	MultiDocumentFileCollection()

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Closes the collection, any files that may be open.
boolean	endOfCollection() Returns true if the end of the collection has been reached
protected void	extractCharset()
abstract Document	getDocument() Get the document object representing the current document.
boolean	hasNext() Check whether it is the last document in the collection
protected void	loadDocumentClass() Loads the class that will supply all documents for this Collection.
Document	next() Return the next document
abstract boolean	nextDocument() Move the collection to the start of the next document.
protected void	openNewFile()
protected boolean	openNextFile() Opens the next document from the collection specification.
protected void	readCollectionSpec(String CollectionSpecFilename) read in the collection.spec
void	reset() Resets the Collection iterator to the start of the collection.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static final org.slf4j.Logger logger

logger for this class

documentsInThisFile

protected int documentsInThisFile

Counts the number of documents that have been found in this file.

eoc

protected boolean eoc

are we at the end of the collection?

eof

protected boolean eof

has the end of the current input file been reached?

SkipFile

protected boolean SkipFile

A boolean which is true when a new file is open.

currentFilename

protected String currentFilename

Filename of current file

forceUTF8

protected final boolean forceUTF8

should UTF8 encoding be assumed?

is

protected InputStream is

the input stream of the current input file

DocProperties

protected Map<String,String> DocProperties

properties for the current document

FilesToProcess

protected ArrayList<String> FilesToProcess

The list of files to process.

FileNumber

protected int FileNumber

The index in the FilesToProcess of the currently processed file.

desiredEncoding

protected String desiredEncoding

Encoding to be used to open all files.

documentClass

protected Class<? extends Document> documentClass

Class to use for all documents parsed by this class

tokeniser

protected Tokeniser tokeniser

Tokeniser to use for all documents parsed by this class

Constructor Detail

MultiDocumentFileCollection

protected MultiDocumentFileCollection()

Method Detail

getDocument

public abstract Document getDocument()

Description copied from interface: Collection

Get the document object representing the current document.

Specified by:

getDocument in interface Collection

Returns:

Document the current document;

loadDocumentClass

protected void loadDocumentClass()

Loads the class that will supply all documents for this Collection. Set by property trec.document.class

hasNext

public boolean hasNext()

Check whether it is the last document in the collection

Returns:

boolean

next

public Document next()

Return the next document

Returns:

next document

close

public void close()

Closes the collection, any files that may be open.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

endOfCollection

public boolean endOfCollection()

Returns true if the end of the collection has been reached

Specified by:

endOfCollection in interface Collection

Returns:

boolean true if the end of collection has been reached, otherwise it returns false.

openNextFile

protected boolean openNextFile()
                        throws IOException

Opens the next document from the collection specification.

Returns:

boolean true if the file was opened successufully. If there are no more files to open, it returns false.

Throws:

IOException - if there is an exception while opening the collection files.

extractCharset

protected void extractCharset()

openNewFile

protected void openNewFile()
                    throws Exception

Throws:

Exception

nextDocument

public abstract boolean nextDocument()

Move the collection to the start of the next document.

Specified by:

nextDocument in interface Collection

Returns:

boolean true if there exists another document in the collection, otherwise it returns false.

readCollectionSpec

protected void readCollectionSpec(String CollectionSpecFilename)

read in the collection.spec

reset

public void reset()

Resets the Collection iterator to the start of the collection.

Specified by:

reset in interface Collection