[antlr-interest] [v3] not including text in token. Still possible?

[Kay Roepke]

On 7. Feb 2006, at 0:50 Uhr, Terence Parr wrote:

>> It sure would, but wouldn't this mean we'd always create a buffer  
>> and copy the string? I'd rather not lose the ability to use the  
>> indexing into the input
>> buffer.
>
> Well, you need a pure flat simple string for the overall token you  
> are creating.  You need to copy from the buffer anyway into the new  
> stirng, right?

Oh, I was talking about the case where I don't have any bangs in the  
rule. But from entering the rule I cannot look forward onto all the  
atoms within
the rule, can I? (Or rather, the tree walker isn't doing that.)
So in the general case without any bangs, I don't want to create any  
string. I want to rely on the indices into the input buffer.

>> Just recording the indices and copy to buffer once I see a bang  
>> and pick up after that was the first idea I had...
>> I could probably set some flag if I've seen a bang and only then  
>> really copy the buffer. Yeah, that might work...
>
> True, but messier I suppose.  Perhaps we should think about the  
> common case like removing the quotes off of string and char  
> literals...how complicated do you want the average case to be...

As straightforward as possible ;) The first thing that should would  
work are the literals, I agree. The rest should be quite similar,  
though.

-k