[antlr-interest] Manipulating lexer text output

[Jim Idle]

Well, perhaps, but the lexer is just looking at one character at a time
anyway and does not produce a string with that stuff in it unless you
ask for it - if you don't then you just get a token that says where the
start and stop is. So it isn't overly inefficient and if you set the
text explicitly then there is probably little difference in execution
time.

Jim



-----Original Message-----
From: Gavin Lambert [mailto:antlr at mirality.co.nz] 
Sent: Tuesday, April 03, 2007 2:00 PM
To: Jim Idle; antlr-interest at antlr.org
Subject: RE: [antlr-interest] Manipulating lexer text output

At 06:54 4/04/2007, Jim Idle wrote:
 >Yeah - this has come up a lot and really the solution at the 
moment
 >is to do this in the parser. However you could at least do this 
in
 >the STRING rule with a call to a small function that removed 
this stuff
 >and then emit the token with the result as the text.

That'd still require double-parsing the input though (once via 
ANTLR's normal processing, and once after the fact to resolve 
escape sequences).  Which makes it no better than doing it outside 
ANTLR entirely.