org.terrier.indexing.tokenisation

Class IdentityTokeniser

java.lang.Object

org.terrier.indexing.tokenisation.Tokeniser

org.terrier.indexing.tokenisation.IdentityTokeniser

public class IdentityTokeniser
extends Tokeniser

A Tokeniser implementation that returns the input as is.

Author:

Craig Macdonald

Field Summary

Fields inherited from class org.terrier.indexing.tokenisation.Tokeniser

EMPTY_STREAM

Constructor Summary

Constructors

Constructor and Description
IdentityTokeniser()

Method Summary

Methods

Modifier and Type	Method and Description
TokenStream	tokenise(Reader reader) Tokenises the text obtained from the specified reader.

Methods inherited from class org.terrier.indexing.tokenisation.Tokeniser

getTokeniser, getTokens

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IdentityTokeniser

public IdentityTokeniser()

Method Detail

tokenise

public TokenStream tokenise(Reader reader)

Description copied from class: Tokeniser

Tokenises the text obtained from the specified reader.

Specified by:

tokenise in class Tokeniser

Parameters:

reader - Stream of text to be tokenised

Returns:

a TokenStream of the tokens found in the text.