[antlr-interest] Multiple lexer tokens per rule
Junkman
j at junkwallah.org
Fri Jun 4 14:16:31 PDT 2010
Ken Williams wrote:
>
>
> On 6/4/10 2:53 PM, "Jim Idle" <jimi at temporal-wave.com> wrote:
>
>> You need to use a collection that gives out the entries in the order they were
>> added:
>>
>> http://java.sun.com/docs/books/tutorial/collections/interfaces/queue.html
>
> I don't think that's the issue here, there's only ever 1 element in the
> queue. It's not getting mixed up with other elements in the queue, it's
> getting mixed up with the token held by the superclass.
>
>
The way nextToken() is overriden, it first returns the token matched by
the rule, and subsequently any additional queued token before matching a
new token in the input stream.
Lexer.emit(x) sets x as the token matched by the rule.
That's why the token emitted with Lexer.emit(Token) is returned before
any queued token(s).
More information about the antlr-interest
mailing list