[antlr-interest] [v3] not including text in token. Still possible?
parrt at cs.usfca.edu
Mon Feb 6 10:39:08 PST 2006
On Feb 6, 2006, at 4:34 AM, Kay Roepke wrote:
>> Hi! I haven't figured out to make it do that for v3 yet. I don't
>> create strings for a token (just indexes into the char buffer) so
>> it's hard to do modifications. I definitely need it though.
> Hi Ter!
> I'd like to give this one a try just for the sake of it. It really
> doesn't matter to me whether it'll end up in ANTLR or not. I just
> like to
> try to implement it to familiarize myself with the source.
> As far as I can tell, I would need to have a look at antlr.g (to
> see if it is correctly recognizing the '!') and then add the
> supporting code
> to codegen.g. After that I have to figure out how to do the actual
> codegeneration in Java.stg.
> Is that correct?
All of the grammars should be already to go. I think the code
generation templates are the only problem. you need to modify the
code generator grammar So that it pays attention of the ! for
lexers. I have already anticipated this problem and added a text
pointer in the common token. The getText method returns the text
pointer if not null else it looks for the indices into the text buffer.
Any rule that has a bang modifier must Create a local string to fill.
Code must be inserted after the match routines to add the matched
char to the local char buffer. I suppose that the emit method must
be altered to accept a string argument representing the text for the
More information about the antlr-interest