[antlr-interest] Multiple lexer tokens per rule

Junkman j at junkwallah.org
Fri Jun 4 14:16:31 PDT 2010


Ken Williams wrote:
> 
> 
> On 6/4/10 2:53 PM, "Jim Idle" <jimi at temporal-wave.com> wrote:
> 
>> You need to use a collection that gives out the entries in the order they were
>> added:
>>
>> http://java.sun.com/docs/books/tutorial/collections/interfaces/queue.html
> 
> I don't think that's the issue here, there's only ever 1 element in the
> queue.  It's not getting mixed up with other elements in the queue, it's
> getting mixed up with the token held by the superclass.
> 
> 

The way nextToken() is overriden, it first returns the token matched by
the rule, and subsequently any additional queued token before matching a
new token in the input stream.

Lexer.emit(x) sets x as the token matched by the rule.

That's why the token emitted with Lexer.emit(Token) is returned before
any queued token(s).




More information about the antlr-interest mailing list