[antlr-interest] Antlr 3 Lexer problem

[Gavin Lambert]

At 09:06 27/06/2007, Geoffrey Zhu wrote:
 >The syntactic predicate does not seem to work. The lexer chokes 
on
 >exactly the same location 'c' if I pass in "( security".
 >
 >In mTokens() it still looks ahead only one step to determine 
what
 >should e the next token.

I think this is another occurrence of the case that Ter claims is 
by design, but myself and a few others would like to be different: 
the lexer doesn't do backtracking, it simply fails with 
NoViableAltExceptions (or the equivalent) -- even when the parent 
grammar does do backtracking.  Basically once it enters a 
particular token it's going to either match that token or cause an 
error; it won't go back and pick a different token.

It is usually possible to work around this by jumping through 
hoops backwards with your eyes closed (aka reordering rules and/or 
juggling predicates, parentheses, and fragment rules until 
something clicks, or just giving up and moving some of it over to 
the parser), but it's definitely not intuitive.