org.terrier.indexing.tokenisation
Class TokenStream

java.lang.Object
  org.terrier.indexing.tokenisation.TokenStream

All Implemented Interfaces:

java.util.Iterator<java.lang.String>

public abstract class TokenStream
extends java.lang.Object
implements java.util.Iterator<java.lang.String>

Represents a stream of tokens found by a tokeniser. It is of note that a TokenStream may return null for a next() method, even if hasNext() previously returned true.

Since:

3.5

Author:

Craig Macdonald

Constructor Summary

TokenStream()

Method Summary

void

remove()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

hasNext, next

Constructor Detail

TokenStream

public TokenStream()

Method Detail

remove

public void remove()

Specified by:

remove in interface java.util.Iterator<java.lang.String>