[antlr-interest] [v3] not including text in token. Still possible?
Terence Parr
parrt at cs.usfca.edu
Mon Feb 6 16:44:59 PST 2006
On Feb 6, 2006, at 4:36 PM, Kay Roepke wrote:
>
> On 7. Feb 2006, at 0:50 Uhr, Terence Parr wrote:
>
>>> It sure would, but wouldn't this mean we'd always create a buffer
>>> and copy the string? I'd rather not lose the ability to use the
>>> indexing into the input
>>> buffer.
>>
>> Well, you need a pure flat simple string for the overall token you
>> are creating. You need to copy from the buffer anyway into the
>> new stirng, right?
>
> Oh, I was talking about the case where I don't have any bangs in
> the rule. But from entering the rule I cannot look forward onto all
> the atoms within
> the rule, can I? (Or rather, the tree walker isn't doing that.)
If there are no bangs lexically present in the rule, don't generate
the special code. :)
> So in the general case without any bangs, I don't want to create
> any string. I want to rely on the indices into the input buffer.
Yep. Only do new stuff if you see a !. Even if it's in a (..)? block.
>>> Just recording the indices and copy to buffer once I see a bang
>>> and pick up after that was the first idea I had...
>>> I could probably set some flag if I've seen a bang and only then
>>> really copy the buffer. Yeah, that might work...
>>
>> True, but messier I suppose. Perhaps we should think about the
>> common case like removing the quotes off of string and char
>> literals...how complicated do you want the average case to be...
>
> As straightforward as possible ;) The first thing that should would
> work are the literals, I agree. The rest should be quite similar,
> though.
:)
Ter
More information about the antlr-interest
mailing list