Uses of Class
org.terrier.indexing.tokenisation.TokenStream

Packages that use TokenStream

org.terrier.indexing	Provides classes and interfaces related to the indexing of documents.
org.terrier.indexing.tokenisation	Provides classes related to the tokenisation of documents.

Uses of TokenStream in org.terrier.indexing

Fields in org.terrier.indexing declared as TokenStream

protected TokenStream	TaggedDocument.currentTokenStream
protected TokenStream	FileDocument.tokenStream

Uses of TokenStream in org.terrier.indexing.tokenisation

Fields in org.terrier.indexing.tokenisation declared as TokenStream

static TokenStream

Tokeniser.EMPTY_STREAM empty stream

Methods in org.terrier.indexing.tokenisation that return TokenStream

TokenStream	UTFTokeniser.tokenise(java.io.Reader reader)
abstract TokenStream	Tokeniser.tokenise(java.io.Reader reader) Tokenises the text obtained from the specified reader.
TokenStream	IdentityTokeniser.tokenise(java.io.Reader reader)
TokenStream	EnglishTokeniser.tokenise(java.io.Reader reader)