[antlr-interest] [v3] not including text in token. Still possible?

Terence Parr parrt at cs.usfca.edu
Mon Feb 6 10:39:08 PST 2006


On Feb 6, 2006, at 4:34 AM, Kay Roepke wrote:
>> Hi!  I haven't figured out to make it do that for v3 yet.  I don't  
>> create strings for a token (just indexes into the char buffer) so  
>> it's hard to do modifications.  I definitely need it though.
>
> Hi Ter!

Howdy do!

> I'd like to give this one a try just for the sake of it. It really  
> doesn't matter to me whether it'll end up in ANTLR or not. I just  
> like to
> try to implement it to familiarize myself with the source.
> As far as I can tell, I would need to have a look at antlr.g (to  
> see if it is correctly recognizing the '!') and then add the  
> supporting code
> to codegen.g. After that I have to figure out how to do the actual  
> codegeneration in Java.stg.
> Is that correct?

All of the grammars should be already to go. I think the code  
generation templates are the only problem.   you need to modify the  
code generator grammar So that it pays attention of the !  for  
lexers.  I have already anticipated this problem and added a text  
pointer in the common token.  The getText  method returns the text  
pointer if not null else it looks for the indices into the text buffer.

Any rule that has a bang modifier must Create a local string to fill.  
Code must be inserted after the match routines to add the matched  
char to the local char buffer.  I suppose that the emit method must  
be altered to accept a string argument representing the text for the  
token.

Ter


More information about the antlr-interest mailing list