[antlr-interest] Lexer bug?

Clifford Heath clifford.heath at gmail.com
Sun Oct 21 17:11:30 PDT 2007


Austin Hastings wrote:
> You're right. I looked at your definition of NUMBER and just assumed you 
> were using the common one. It looks like a bug.
> 
> In fact, (some time later) I'm looking at the generated code now with 
> new disrespect. The tokenizer is doing a minimal look-ahead and then 
> committing to a token - when it sees '1' in your 10..20 example, it 
> commits to a NUMBER. When it comes to '.' it commits to FRACTION. There 
> doesn't appear to be any consideration that one path might fail and 
> another be chosen.

I feared this must be what was happening...
And yet the DIGIT+ path must fail, but no error is reported.
So there's *two* errors in the generated lexer, one where it
takes the wrong path, and one where it doesn't report the error
it sees.

This second error must be affecting other cases of invalid input as well...?

Clifford Heath.



More information about the antlr-interest mailing list