org.terrier.indexing

Class FlatJSONDocument

java.lang.Object

org.terrier.indexing.FlatJSONDocument

All Implemented Interfaces:

Document

public class FlatJSONDocument
extends Object
implements Document

This is a Terrier Document implementation of a document stored in JSON format. It assumes that a single JSON document has at least a single attribute called 'text' that contains the text of the document. Fields: This implementation supports a single field named 'TEXT' by default. FieldTags.process is a comma delimited list of properties to use as fields. Meta-Data: During the parsing process, the properties of each FlatJSONDocument is decorated with document meta-data. This decoration process is performed by 'flattening' the layered structure of the JSON object and its sub-attributes into individual properties. For property naming, attributes in different layers are connected with a dot '.', e.g. user.name

Since:

5.1

Author:

Richard McCreadie and Saul Vargas

Field Summary

Fields

Modifier and Type	Field and Description
protected int	fieldIndex
protected List<String>	fieldQueue
protected String[]	fieldsToProcess
protected Map<String,String>	properties
protected int	remainingTokens
protected int	tokenIndex
protected Tokeniser	tokenizer
String[][]	tokens

Constructor Summary

Constructors

Constructor and Description
FlatJSONDocument(com.google.gson.JsonObject json)
FlatJSONDocument(String rawJson)

Method Summary

Modifier and Type	Method and Description
boolean	endOfDocument() Returns true when the end of the document has been reached, and there are no other terms to be retrieved from it.
Map<String,String>	getAllProperties() Returns the underlying map of all the properties defined by this Document.
Set<String>	getFields() Returns a list of the fields the current term appears in.
String	getNextTerm() Gets the next term of the document.
String	getProperty(String name) Allows access to a named property of the Document.
Reader	getReader() Returns a Reader object so client code can tokenise the document or deal with the document itself.
protected void	initalize(String rawJson)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

properties

protected Map<String,String> properties

tokenizer

protected Tokeniser tokenizer

tokens

public String[][] tokens

fieldQueue

protected List<String> fieldQueue

fieldsToProcess

protected String[] fieldsToProcess

fieldIndex

protected int fieldIndex

tokenIndex

protected int tokenIndex

remainingTokens

protected int remainingTokens

Constructor Detail

FlatJSONDocument

public FlatJSONDocument(com.google.gson.JsonObject json)

FlatJSONDocument

public FlatJSONDocument(String rawJson)
                 throws com.fasterxml.jackson.core.JsonParseException,
                        com.fasterxml.jackson.databind.JsonMappingException,
                        IOException

Throws:

com.fasterxml.jackson.core.JsonParseException

com.fasterxml.jackson.databind.JsonMappingException

IOException

Method Detail

initalize

protected void initalize(String rawJson)

endOfDocument

public boolean endOfDocument()

Description copied from interface: Document

Returns true when the end of the document has been reached, and there are no other terms to be retrieved from it.

Specified by:

endOfDocument in interface Document

Returns:

boolean true if there are no more terms in the document, otherwise it returns false.

getAllProperties

public Map<String,String> getAllProperties()

Description copied from interface: Document

Returns the underlying map of all the properties defined by this Document.

Specified by:

getAllProperties in interface Document

getFields

public Set<String> getFields()

Description copied from interface: Document

Returns a list of the fields the current term appears in.

Specified by:

getFields in interface Document

Returns:

HashSet a set of the terms that the current term appears in.

getNextTerm

public String getNextTerm()

Description copied from interface: Document

Gets the next term of the document. NB:Null string returned from getNextTerm() should be ignored. They do not signify the lack of any more terms. endOfDocument() should be used to check that.

Specified by:

getNextTerm in interface Document

Returns:

String the next term of the document. Null returns should be ignored.

getProperty

public String getProperty(String name)

Description copied from interface: Document

Allows access to a named property of the Document. Examples might be URL, filename etc.

Specified by:

getProperty in interface Document

Parameters:

name - Name of the property. It is suggested, but not required that this name should not be case insensitive.

getReader

public Reader getReader()

Description copied from interface: Document

Returns a Reader object so client code can tokenise the document or deal with the document itself. Examples might be extracting URLs, language detection.

Specified by:

getReader in interface Document