org.terrier.indexing.tokenisation
Class IdentityTokeniser

java.lang.Object
  org.terrier.indexing.tokenisation.Tokeniser
      org.terrier.indexing.tokenisation.IdentityTokeniser

public class IdentityTokeniser
extends Tokeniser

A Tokeniser implementation that returns the input as is.

Author:

Craig Macdonald

Field Summary

Fields inherited from class org.terrier.indexing.tokenisation.Tokeniser

EMPTY_STREAM

Constructor Summary

IdentityTokeniser()

Method Summary

TokenStream

tokenise(java.io.Reader reader) Tokenises the text obtained from the specified reader.

Methods inherited from class org.terrier.indexing.tokenisation.Tokeniser

getTokeniser, getTokens

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IdentityTokeniser

public IdentityTokeniser()

Method Detail

tokenise

public TokenStream tokenise(java.io.Reader reader)

Description copied from class: Tokeniser

Tokenises the text obtained from the specified reader.

Specified by:

tokenise in class Tokeniser

Parameters:

reader - Stream of text to be tokenised

Returns:

a TokenStream of the tokens found in the text.