org.cdlib.xtf.util

Class CheckingTokenStream

Object

TokenStream

CheckingTokenStream

public class CheckingTokenStream
extends TokenStream

Used for debugging optimized FastTokenStream, this class checks the main TokenStream against a reference one for equality. Any difference is flagged with an assertion failure.

Author:

Martin Haye

Field Summary

Fields

Modifier and Type	Field and Description
private static String	ACRONYM_TYPE Token type for acronyms
private static String	APOSTROPHE_TYPE Token type for words containing apostrophes
(package private) TokenStream	main Main token stream that is being checked
(package private) TokenStream	ref Reference stream to check the main one against

Constructor Summary

Constructors

Constructor and Description
CheckingTokenStream(TokenStream main, TokenStream ref) Construct a CheckingTokenStream

Method Summary

Modifier and Type	Method and Description
void	close() Close the token stream
private String	idType(String type) Map the type to apostrophe, acronym, or other
Token	next() Get the next token from the main stream.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

main

TokenStream main

Main token stream that is being checked

ref

TokenStream ref

Reference stream to check the main one against

APOSTROPHE_TYPE

private static final String APOSTROPHE_TYPE

Token type for words containing apostrophes

ACRONYM_TYPE

private static final String ACRONYM_TYPE

Token type for acronyms

Constructor Detail

CheckingTokenStream

public CheckingTokenStream(TokenStream main,
                           TokenStream ref)

Construct a CheckingTokenStream

Method Detail

next

public Token next()
           throws IOException

Get the next token from the main stream. Checks that this token matches the next one in the reference stream.

Specified by:

next in class TokenStream

Throws:

IOException

idType

private String idType(String type)

Map the type to apostrophe, acronym, or other

close

public void close()
           throws IOException

Close the token stream

Overrides:

close in class TokenStream

Throws:

IOException