org.terrier.indexing.tokenisation

Class TokenStream

java.lang.Object

org.terrier.indexing.tokenisation.TokenStream

All Implemented Interfaces:

Iterator<String>

public abstract class TokenStream
extends Object
implements Iterator<String>

Represents a stream of tokens found by a tokeniser. It is of note that a TokenStream may return null for a next() method, even if hasNext() previously returned true.

Since:

3.5

Author:

Craig Macdonald

Constructor Summary

Constructors

Constructor and Description
TokenStream()

Method Summary

Methods

Modifier and Type	Method and Description
void	remove()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

hasNext, next

Constructor Detail

TokenStream

public TokenStream()

Method Detail

remove

public void remove()

Specified by:

remove in interface Iterator<String>