[antlr-interest] Problem with simple tokens

[Markus Stoeger]

Gavin Lambert schrieb:
> In this case, what's happening is that the input 'foo' could match 
> either Foo or Identifier; by itself ANTLR will choose Foo, since it's 
> listed first -- but when given the input 'foo.', this could either be 
> "Foo '.'" or "Identifier" (admittedly not a complete Identifier, but 
> it doesn't realise that yet), so it'll pick Identifier since it 
> consumes more of the input in one go.
Thanks for the explanation. I have also read some of your other messages 
concerning the lexer.. very helpful.. some of them should make it into 
the docs. ANTLR's lexer's behaviour still seems strange to me in that 
case.. I'll have to think about that some more and I'll try to change my 
grammar definition to avoid such cases.

Max