Uses of Class
org.terrier.indexing.tokenisation.TokenStream

Packages that use TokenStream

Package	Description
org.terrier.indexing	Provides classes and interfaces related to the indexing of documents.
org.terrier.indexing.tokenisation	Provides classes related to the tokenisation of documents.

Uses of TokenStream in org.terrier.indexing

Fields in org.terrier.indexing declared as TokenStream

Modifier and Type	Field and Description
protected TokenStream	TaggedDocument.currentTokenStream
protected TokenStream	FileDocument.tokenStream

Uses of TokenStream in org.terrier.indexing.tokenisation

Fields in org.terrier.indexing.tokenisation declared as TokenStream

Modifier and Type	Field and Description
static TokenStream	Tokeniser.EMPTY_STREAM empty stream

Methods in org.terrier.indexing.tokenisation that return TokenStream

Modifier and Type	Method and Description
TokenStream	UTFTokeniser.tokenise(Reader reader)
TokenStream	UTFTwitterTokeniser.tokenise(Reader reader)
TokenStream	IdentityTokeniser.tokenise(Reader reader)
abstract TokenStream	Tokeniser.tokenise(Reader reader) Tokenises the text obtained from the specified reader.
TokenStream	EnglishTokeniser.tokenise(Reader reader)