[antlr-interest] Best practice to handle Lexer backtracking demand

[Gerald Rosenberg]

------ Original Message (Saturday, August 14, 2010 2:13:47 
AM) From: Joachim Schrod ------
Subject: [antlr-interest] Best practice to handle Lexer backtracking demand

Doubt that there is any one true best practice.  A truism that may help 
is "try not to do too much in the lexer."

Given that your valid input text is pretty much defined by

>    CHAR : . ;
>

likely best to defer key word matching to the parser

>    name : n=text ( { helper.isKeyword($n) }? text )?  ;
>    text : CHAR+ ;
>
>    CHAR : . ;
and provide for the helper in the parser::members block.
> Using syntactic predicates? I tried that but did not succeed.
> AFAIU, they are a parser feature, but I'd need them in the lexer.
No, they work in the lexer as well.  It is just that the lexer defaults 
effectively k=1, so the predicated alternatives need to be specified.  
Or, use an embedded options block to push up the value of k.