[antlr-interest] Enhancement request for identifying imaginary tokens
Hardy, Stephen
Stephen.Hardy at digi.com
Fri Dec 14 09:47:49 PST 2007
As I said, I have probably missed something obvious, but what I meant
was that
options { output = AST; IdentifyImaginaryTokensWithStart = true; }
tokens { FOO; }
foo: bar baz -> ^(FOO bar baz);
would be the same as what must now be done explicitly for each imaginary
node i.e.
options { output = AST; }
tokens { FOO; }
foo: bar baz -> ^(FOO[$start] bar baz);
Currently, it seems that without the [] 'constructor parameter' the
imaginary tokens do not get any line/colum number information. Thus, if
a tree walker gets confused and tries to print an imaginary token in its
error message, then the line/col info is always 0:0. This makes it
harder to debug, which is why I went through my grammar and explicitly
assigned the $start token to each imaginary token created.
(And now, having said that, maybe it is already the default and it was
something else that I did wrong which made me think otherwise; after
all, I do create a few extra nodes programmatically and maybe it was
these which exhibited the line/col problem. If so, I apologize for
wasting your time.)
Regards,
SJH
> -----Original Message-----
> From: Terence Parr [mailto:parrt at cs.usfca.edu]
> Sent: Friday, December 14, 2007 9:32 AM
> To: Hardy, Stephen
> Cc: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] Enhancement request for
> identifying imaginary tokens
>
> Hi Stephen, i'm not sure what you are asking for. You must say what
> root node to create. How would antlr know what you want?
>
> thanks,
>
> Ter
> On Dec 13, 2007, at 2:47 PM, Hardy, Stephen wrote:
>
> > Maybe I missed something, but it would be nice if there was an
> > option to
> > automatically "identify" all imaginary tokens. Currently, this
> > requires
> > explicit identification as per the following example:
> >
> > struct_declarator
> > : declarator (':' constant_expression)?
> > -> ^(STRUCT_DECLARATOR[$start] declarator
> constant_expression?)
> > | ':' constant_expression
> > -> ^(STRUCT_DECLARATOR_ANON_BITFIELD[$start]
> > constant_expression)
> > ;
> >
> >
> > Admittedly, it's not all that difficult to put the
> > MY_IMAGINARY_TOKEN[$start] construct everywhere in the tree writing
> > part, but it does make the code less readable. Having an
> option to do
> > this by default would be very handy when debugging tree walkers.
> > Something like
> >
> > options {
> > IdentifyImaginaryTokensWithStart = true;
> > }
> >
> > Regards,
> > SJH
> >
>
>
More information about the antlr-interest
mailing list