[antlr-interest] finally added UnbufferedTokenStream

[Terence Parr]

To build ST v4's insanely complicated lexer, I couldn't get around an  
on-demand (unbuffered) token stream.  Note that I don't have an  
unbuffered char stream yet but given LookaheadStream<T>, it should be  
just as easy.  Code for token stream is tiny; see below.  It seems to  
work so far.

Ter

add //depot/code/antlr/main/runtime/Java/src/main/java/org/antlr/ 
runtime/UnbufferedTokenStream.java#1
edit //depot/code/antlr/main/runtime/Java/src/main/java/org/antlr/ 
runtime/misc/LookaheadStream.java#4
edit //depot/code/antlr/main/tool/CHANGES.txt#18
Change 6294 submitted.

package org.antlr.runtime;

import org.antlr.runtime.misc.LookaheadStream;

import java.util.Map;
import java.util.Set;

public class UnbufferedTokenStream extends LookaheadStream<Token>  
implements TokenStream {
	protected TokenSource tokenSource;
	protected int tokenIndex = 0;

	public UnbufferedTokenStream(TokenSource tokenSource) {
		super(Token.EOF_TOKEN);
		this.tokenSource = tokenSource;
	}

	public Token nextElement() {
		Token t = tokenSource.nextToken();
		t.setTokenIndex(tokenIndex++);
		return t;
	}

	public TokenSource getTokenSource() { return tokenSource; }

	public String toString(int start, int stop) { return "n/a"; }

	public String toString(Token start, Token stop) { return "n/a"; }

	public int LA(int i) { return LT(i).getType(); }

	public String getSourceName() {	return tokenSource.getSourceName();	}
}