Class IdentityTokeniser

  • All Implemented Interfaces:

    public class IdentityTokeniser
    extends Tokeniser
    A Tokeniser implementation that returns the input as is.
    Craig Macdonald
    See Also:
    Serialized Form
    • Constructor Detail

      • IdentityTokeniser

        public IdentityTokeniser()
    • Method Detail

      • tokenise

        public TokenStream tokenise​( reader)
        Description copied from class: Tokeniser
        Tokenises the text obtained from the specified reader.
        Specified by:
        tokenise in class Tokeniser
        reader - Stream of text to be tokenised
        a TokenStream of the tokens found in the text.