[antlr-interest] Discarding characters in lexer

Gavin Lambert antlr at mirality.co.nz
Thu Mar 27 12:33:12 PDT 2008


At 04:02 28/03/2008, Andrey Timoshenkov wrote:
 >I have a possibly quick question about the lexical analyzer. In
 >version 2 there was a '!' operator that allowed to discard
 >characters from token text like this:
 >
 >STRING: '"'! ('"' '"'! | ~('"') )* '"'! ;
 >
 >It seems that it has gone in version 3. Is it possible to 
achieve
 >the same result in version 3?

In v3, the ! operator only works in the parser (and affects AST 
generation).  You can't use it in the lexer any more.

To achieve the result you want now, you'll need to call $setText 
at the end of the rule (which means you'll have to rescan the 
string outside of ANTLR to convert escape sequences and the 
like).  Bit of a hassle, I agree, but that's the only option for 
now.



More information about the antlr-interest mailing list