[antlr-interest] [v3] not including text in token. Still possible?

Terence Parr parrt at cs.usfca.edu
Mon Feb 6 16:44:59 PST 2006


On Feb 6, 2006, at 4:36 PM, Kay Roepke wrote:

>
> On 7. Feb 2006, at 0:50 Uhr, Terence Parr wrote:
>
>>> It sure would, but wouldn't this mean we'd always create a buffer  
>>> and copy the string? I'd rather not lose the ability to use the  
>>> indexing into the input
>>> buffer.
>>
>> Well, you need a pure flat simple string for the overall token you  
>> are creating.  You need to copy from the buffer anyway into the  
>> new stirng, right?
>
> Oh, I was talking about the case where I don't have any bangs in  
> the rule. But from entering the rule I cannot look forward onto all  
> the atoms within
> the rule, can I? (Or rather, the tree walker isn't doing that.)

If there are no bangs lexically present in the rule, don't generate  
the special code. :)

> So in the general case without any bangs, I don't want to create  
> any string. I want to rely on the indices into the input buffer.

Yep.  Only do new stuff if you see a !.  Even if it's in a (..)? block.

>>> Just recording the indices and copy to buffer once I see a bang  
>>> and pick up after that was the first idea I had...
>>> I could probably set some flag if I've seen a bang and only then  
>>> really copy the buffer. Yeah, that might work...
>>
>> True, but messier I suppose.  Perhaps we should think about the  
>> common case like removing the quotes off of string and char  
>> literals...how complicated do you want the average case to be...
>
> As straightforward as possible ;) The first thing that should would  
> work are the literals, I agree. The rest should be quite similar,  
> though.

:)

Ter



More information about the antlr-interest mailing list