[antlr-interest] Enhancement request for identifying imaginary tokens

[Hardy, Stephen]

As I said, I have probably missed something obvious, but what I meant
was that

options { output = AST; IdentifyImaginaryTokensWithStart = true; }
tokens { FOO; }
foo:  bar baz  -> ^(FOO bar baz);

would be the same as what must now be done explicitly for each imaginary
node i.e.

options { output = AST; }
tokens { FOO; }
foo:  bar baz  -> ^(FOO[$start] bar baz);

Currently, it seems that without the [] 'constructor parameter' the
imaginary tokens do not get any line/colum number information.  Thus, if
a tree walker gets confused and tries to print an imaginary token in its
error message, then the line/col info is always 0:0.  This makes it
harder to debug, which is why I went through my grammar and explicitly
assigned the $start token to each imaginary token created.

(And now, having said that, maybe it is already the default and it was
something else that I did wrong which made me think otherwise; after
all, I do create a few extra nodes programmatically and maybe it was
these which exhibited the line/col problem.  If so, I apologize for
wasting your time.)

Regards,
SJH


> -----Original Message-----
> From: Terence Parr [mailto:parrt at cs.usfca.edu] 
> Sent: Friday, December 14, 2007 9:32 AM
> To: Hardy, Stephen
> Cc: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] Enhancement request for 
> identifying imaginary tokens
> 
> Hi Stephen, i'm not sure what you are asking for.  You must say what  
> root node to create.  How would antlr know what you want?
> 
> thanks,
> 
> Ter
> On Dec 13, 2007, at 2:47 PM, Hardy, Stephen wrote:
> 
> > Maybe I missed something, but it would be nice if there was an  
> > option to
> > automatically "identify" all imaginary tokens.  Currently, this  
> > requires
> > explicit identification as per the following example:
> >
> > struct_declarator
> >    : declarator (':' constant_expression)?	
> >        -> ^(STRUCT_DECLARATOR[$start] declarator 
> constant_expression?)
> >    | ':' constant_expression			
> >        -> ^(STRUCT_DECLARATOR_ANON_BITFIELD[$start]
> > constant_expression)
> >    ;
> >
> >
> > Admittedly, it's not all that difficult to put the
> > MY_IMAGINARY_TOKEN[$start] construct everywhere in the tree writing
> > part, but it does make the code less readable.  Having an 
> option to do
> > this by default would be very handy when debugging tree walkers.
> > Something like
> >
> > options {
> >  IdentifyImaginaryTokensWithStart = true;
> > }
> >
> > Regards,
> > SJH
> >
> 
>