[antlr-interest] [v3] not including text in token. Still possible?

Terence Parr parrt at cs.usfca.edu
Mon Feb 6 15:50:23 PST 2006


On Feb 6, 2006, at 3:32 PM, Kay Roepke wrote:

>
> On 7. Feb 2006, at 0:20 Uhr, Terence Parr wrote:
>
>> Hmm...well i used to just record the start index before every  
>> element (token or rule) and then if not a bang, add chars from  
>> saved index to current index into buffer.  If a bang, then don't  
>> add.  Would that work?
>
> It sure would, but wouldn't this mean we'd always create a buffer  
> and copy the string? I'd rather not lose the ability to use the  
> indexing into the input
> buffer.

Well, you need a pure flat simple string for the overall token you  
are creating.  You need to copy from the buffer anyway into the new  
stirng, right?

> Just recording the indices and copy to buffer once I see a bang and  
> pick up after that was the first idea I had...
> I could probably set some flag if I've seen a bang and only then  
> really copy the buffer. Yeah, that might work...

True, but messier I suppose.  Perhaps we should think about the  
common case like removing the quotes off of string and char  
literals...how complicated do you want the average case to be...

Ter



More information about the antlr-interest mailing list