[antlr-interest] Manipulating text in the lexer

[Gavin Lambert]

At 04:48 27/02/2009, Sam Barnett-Cormack wrote:
 >http://www.antlr.org/blog/antlr3/lexical.tml suggests that it's
 >no longer possible to alter the content of a token away from
 >what's on the input at all.

That isn't true.  You can use setText or $text = "..." to change 
the text of a token just fine.

What's probably biting you, though, is that this only works from 
the top-level lexer rule.  You can't change the text of a fragment 
(or rather, you can, but it won't end up being used in the final 
token).

 >fragment
 >CSTRINGNL : WSNONL* NL WSNONL* {setText("");};

So this setText will have absolutely no effect.

 >CSTRING : '"' ((CSTRINGNL)=> CSTRINGNL | '""' | ~'"') '"';

But if you called it here, then it would work.