[antlr-interest] Lexer lookahead overoptimizes

[Terence Parr]

On Apr 12, 2007, at 2:40 PM, shmuel siegel wrote:
> That leaves me confused as to what "?" means on a sequence. I  
> thought that it means optional sequence but find that here it only  
> means optional if the beginning of the sequence is not matched. It  
> surprises me that it is possible for an optional sequence to throw  
> a recognition exception. Why is the '\u00d7' optional but not the  
> full sequence?

If it sees the first item, the predictor does not need to see the  
next one.  It can't see past a token boundary.

Ter