[antlr-interest] Multiple lexer tokens per rule

[Ken Williams]

On 6/4/10 4:16 PM, "Junkman" <j at junkwallah.org> wrote:
> 
> The way nextToken() is overriden, it first returns the token matched by
> the rule, and subsequently any additional queued token before matching a
> new token in the input stream.

Maybe I'm being dense here, but I don't think that's what it's doing:

    public Token nextToken() {
        return tokenQueue.isEmpty() ? super.nextToken() : tokenQueue.poll();
    }

If tokenQueue() is non-empty, it always uses it.  On the *next* invocation,
when it's empty, it will call super.nextToken().


-- 
Ken Williams
Sr. Research Scientist
Thomson Reuters
Phone: 651-848-7712
ken.williams at thomsonreuters.com