[antlr-interest] (follow up) setting, altering text in lexer rules

[Kay Roepke]

On 12. Jun 2006, at 23:05 Uhr, Terence Parr wrote:

> Yep, and a simple action with a substring will work; like 1 or 2  
> places in the grammar you want; should be ok to do manually.
>
> setText(getText().substring(0, getText().length()))
>
> or whatever.  Or, just buffer up manually in CHARS.

This is what I do in the ST debugger. I'm using fragment lexer rules  
to grab the interesting bits from a string and then get those
texts in a later action.
The only downside is that this will create (in my case) two  
additional tokens which are discarded at rule exit.
With '!'s this would be easier to do, but the implementation overhead/ 
effort is pretty steep. The initial stab I had at the problem
a couple months ago didn't add too much code, but wasn't nearly  
complete. Especially if you wanted to skip text in the middle of a rule
it meant a lot string copying. Plus it isn't exactly easy to optimize  
for the case that the !s are at the edges.

I think it's better to put this off for later and then try to do a  
non-costly implementation.

-k