org.terrier.indexing

Class TwitterJSONCollection

java.lang.Object

org.terrier.indexing.TwitterJSONCollection

All Implemented Interfaces:

Closeable, AutoCloseable, Collection

public class TwitterJSONCollection
extends Object
implements Collection

This class represents a collection of tweets stored in JSON format. Like TRECCollection, it expects a collection specification containing all of the files to be read. Each file is assumed to be in gzip format, with one tweet per line. The google.gson parser is used to read the tweet JSON. The FlatJSONDocument representation is used.

Since:

4.0

Author:

Richard McCreadie

Field Summary

Fields

Modifier and Type	Field and Description
protected Document	currentDocument The current document
protected String	currentFilename The name of the current file
protected BufferedReader	currentTweetStream The underlying file stream reading tweets from the current file
protected boolean	endOfCollection Have we reached the end of the collection yet?
protected int	FileNumber The index in the FilesToProcess of the currently processed file.
protected List<String>	FilesToProcess The list of files to process.
protected com.google.gson.JsonStreamParser	JSONStream The JSON stream containing the tweets
protected static org.slf4j.Logger	logger logger for this class
protected boolean	SkipFile A boolean which is true when a new file is open.

Constructor Summary

Constructors

Constructor and Description
TwitterJSONCollection()
TwitterJSONCollection(List<String> files, String ignored1, String ignored2, String ignored3)
TwitterJSONCollection(String CollectionSpecFile)
TwitterJSONCollection(String addressCollectionFilename, String ignored1, String ignored2, String ignored3) additional constructors required by TRECIndexing

Method Summary

Modifier and Type	Method and Description
void	addFileToProcess(String JSONFile)
void	close()
boolean	endOfCollection() Returns true if the end of the collection has been reached
Document	getDocument() Get the document object representing the current document.
void	init()
protected void	loadJSON(String file)
boolean	nextDocument() Move the collection to the start of the next document.
boolean	openNextFile() Opens the next document from the collection specification.
protected void	readCollectionSpec(String CollectionSpecFilename)
com.google.gson.JsonObject	readTweet()
void	reset() Resets the Collection iterator to the start of the collection.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static final org.slf4j.Logger logger

logger for this class

FilesToProcess

protected List<String> FilesToProcess

The list of files to process.

SkipFile

protected boolean SkipFile

A boolean which is true when a new file is open.

JSONStream

protected com.google.gson.JsonStreamParser JSONStream

The JSON stream containing the tweets

currentTweetStream

protected BufferedReader currentTweetStream

The underlying file stream reading tweets from the current file

currentDocument

protected Document currentDocument

The current document

currentFilename

protected String currentFilename

The name of the current file

FileNumber

protected int FileNumber

The index in the FilesToProcess of the currently processed file.

endOfCollection

protected boolean endOfCollection

Have we reached the end of the collection yet?

Constructor Detail

TwitterJSONCollection

public TwitterJSONCollection(String CollectionSpecFile)

TwitterJSONCollection

public TwitterJSONCollection()

TwitterJSONCollection

public TwitterJSONCollection(String addressCollectionFilename,
                             String ignored1,
                             String ignored2,
                             String ignored3)

additional constructors required by TRECIndexing

TwitterJSONCollection

public TwitterJSONCollection(List<String> files,
                             String ignored1,
                             String ignored2,
                             String ignored3)

Method Detail

init

public void init()

loadJSON

protected void loadJSON(String file)
                 throws IOException

Throws:

IOException

addFileToProcess

public void addFileToProcess(String JSONFile)

readCollectionSpec

protected void readCollectionSpec(String CollectionSpecFilename)

openNextFile

public boolean openNextFile()
                     throws IOException

Opens the next document from the collection specification.

Returns:

boolean true if the file was opened successufully. If there are no more files to open, it returns false.

Throws:

IOException - if there is an exception while opening the collection files.

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

nextDocument

public boolean nextDocument()

Description copied from interface: Collection

Move the collection to the start of the next document.

Specified by:

nextDocument in interface Collection

Returns:

boolean true if there exists another document in the collection, otherwise it returns false.

readTweet

public com.google.gson.JsonObject readTweet()

getDocument

public Document getDocument()

Description copied from interface: Collection

Get the document object representing the current document.

Specified by:

getDocument in interface Collection

Returns:

Document the current document;

endOfCollection

public boolean endOfCollection()

Description copied from interface: Collection

Returns true if the end of the collection has been reached

Specified by:

endOfCollection in interface Collection

Returns:

boolean true if the end of collection has been reached, otherwise it returns false.

reset

public void reset()

Description copied from interface: Collection

Resets the Collection iterator to the start of the collection.

Specified by:

reset in interface Collection