Uses of Class
org.terrier.indexing.tokenisation.Tokeniser

Packages that use Tokeniser

Package	Description
org.terrier.applications.batchquerying	Contains application-level programs for performing batch-based query operations, such as generating runs for TREC.
org.terrier.indexing	Provides classes and interfaces related to the indexing of documents.
org.terrier.indexing.tokenisation	Provides classes related to the tokenisation of documents.

Uses of Tokeniser in org.terrier.applications.batchquerying

Fields in org.terrier.applications.batchquerying declared as Tokeniser

Modifier and Type	Field	Description
protected Tokeniser	SingleLineTRECQuery.tokeniser

Methods in org.terrier.applications.batchquerying with parameters of type Tokeniser

Modifier and Type	Method	Description
static QuerySource	QuerySourceUtils.create(java.lang.String[] qids, java.lang.String[] qs, Tokeniser tok)

Constructors in org.terrier.applications.batchquerying with parameters of type Tokeniser

Constructor	Description
SingleLineTRECQuery(java.lang.String[] queryfilenames, Tokeniser tokeniser)
SingleLineTRECQuery(java.lang.String queryfilename, Tokeniser tokeniser)

Uses of Tokeniser in org.terrier.indexing

Fields in org.terrier.indexing declared as Tokeniser

Modifier and Type	Field	Description
protected Tokeniser	MultiDocumentFileCollection.tokeniser	Tokeniser to use for all documents parsed by this class
protected Tokeniser	SimpleFileCollection.tokeniser
protected Tokeniser	TaggedDocument.tokeniser
protected Tokeniser	FlatJSONDocument.tokenizer

Methods in org.terrier.indexing with parameters of type Tokeniser

Modifier and Type	Method	Description
static Document	IndexTestUtils.makeDocumentFromText(java.lang.String contents, java.util.Map<java.lang.String,java.lang.String> docProperties, Tokeniser t)

Constructors in org.terrier.indexing with parameters of type Tokeniser

Constructor	Description
FileDocument(java.io.InputStream docStream, java.util.Map<java.lang.String,java.lang.String> docProperties, Tokeniser tok)	Constructs an instance of the FileDocument from the given input stream.
FileDocument(java.io.Reader docReader, java.util.Map<java.lang.String,java.lang.String> docProperties, Tokeniser tok)	create a document for a file
FileDocument(java.lang.String _filename, java.io.InputStream docStream, Tokeniser tok)	create a document for a file
FileDocument(java.lang.String _filename, java.io.Reader docReader, Tokeniser tok)	create a document for a file
MSExcelDocument(java.io.InputStream docStream, java.util.Map<java.lang.String,java.lang.String> docProperties, Tokeniser tok)	Deprecated.
MSExcelDocument(java.lang.String filename, java.io.InputStream docStream, Tokeniser tokeniser)	Deprecated.
MSPowerPointDocument(java.io.InputStream docStream, java.util.Map<java.lang.String,java.lang.String> docProperties, Tokeniser tok)	Deprecated.
MSPowerPointDocument(java.lang.String filename, java.io.InputStream docStream, Tokeniser tokeniser)	Deprecated.
MSWordDocument(java.io.InputStream docStream, java.util.Map<java.lang.String,java.lang.String> docProperties, Tokeniser tok)	Deprecated.
MSWordDocument(java.lang.String filename, java.io.InputStream docStream, Tokeniser tokeniser)	Deprecated.
PDFDocument(java.io.InputStream docStream, java.util.Map<java.lang.String,java.lang.String> docProperties, Tokeniser tok)	Constructs a new PDFDocument
PDFDocument(java.io.Reader docReader, java.util.Map<java.lang.String,java.lang.String> docProperties, Tokeniser tok)	Constructs a new PDFDocument
PDFDocument(java.lang.String filename, java.io.InputStream docStream, Tokeniser tokeniser)	Constructs a new PDFDocument, which will convert the docStream which represents the file to a Document object from which an Indexer can retrieve a stream of terms.
PDFDocument(java.lang.String filename, java.io.Reader docReader, Tokeniser tok)	Constructs a new PDFDocument
POIDocument(java.io.InputStream docStream, java.util.Map<java.lang.String,java.lang.String> docProperties, Tokeniser tok)	Constructs a new MSWordDocument object for the file represented by docStream.
POIDocument(java.lang.String filename, java.io.InputStream docStream, Tokeniser tokeniser)	Constructs a new MSWordDocument object for the file represented by docStream.
TaggedDocument(java.io.InputStream docStream, java.util.Map<java.lang.String,java.lang.String> docProperties, Tokeniser _tokeniser)	Constructs an instance of the class from the given input stream.
TaggedDocument(java.io.InputStream docStream, java.util.Map<java.lang.String,java.lang.String> docProperties, Tokeniser _tokeniser, java.lang.String doctags, java.lang.String exactdoctags, java.lang.String fieldtags)	Constructs an instance of the class from the given input stream.
TaggedDocument(java.io.Reader docReader, java.util.Map<java.lang.String,java.lang.String> docProperties, Tokeniser _tokeniser)	Constructs an instance of the class from the given reader object.

Uses of Tokeniser in org.terrier.indexing.tokenisation

Subclasses of Tokeniser in org.terrier.indexing.tokenisation

Modifier and Type	Class	Description
class	EnglishTokeniser	Tokenises text obtained from a text stream assuming English language.
class	IdentityTokeniser	A Tokeniser implementation that returns the input as is.
class	UTFTokeniser	Tokenises text obtained from a text stream.
class	UTFTwitterTokeniser	A tokeniser designed for use on tweets.

Methods in org.terrier.indexing.tokenisation that return Tokeniser

Modifier and Type	Method	Description
static Tokeniser	Tokeniser.getTokeniser()	Instantiates Tokeniser class named in the tokeniser property.