[antlr-interest] Enhancement request for identifyingimaginarytokens

[Hardy, Stephen]

You're right in that I have slightly overused ITs, however in cases
where there is a logical choice (like the operator in expressions) I
just pull that token to the root of the new sub-tree.  It's when one
gets to higher level constructs of the language where ITs become more
useful, since there is often no reasonable token to use as the root.
For example, one could use the ';' that terminates statements as the
root node for all statement sub-trees, but I think it's nicer to have
distinct ITs for expression statements, if statements, for statements
etc. -- the theory is that this makes tree walkers more efficient, if
not easier to understand.  In many cases the actual ';' or other
delimiting token is not actually part of the rule where one would like
to ref it, since it would not be the most natural way to express the
grammar.


Regards,
SJH

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org 
> [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Terence Parr
> Sent: Friday, December 14, 2007 10:09 AM
> To: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] Enhancement request for 
> identifyingimaginarytokens
> 
> 
> On Dec 14, 2007, at 10:04 AM, Hardy, Stephen wrote:
> 
> > Hi Terence,
> >
> > Understood, but with a large grammar it becomes a bit tedious to
> > explicitly put for all imaginary tokens.  Also, there is often no
> > convenient token to co-opt for the purpose.  I thought being able to
> > specify a default of the "start" token (whatever it may be) would be
> > useful for most folks, provided there was the option to override the
> > token assignment as you showed below, with the usual [] syntax.
> 
> Possibly.  Are you overusing imaginary tokens though?  Seems you  
> create one for every rewrite.  Why not just ref existing token?
> 
> Ter
>