[antlr-interest] What's the motivation of using the "tokens section"?

[Austin Hastings]

The tokens section is used to declare tokens that you have not specified 
elsewhere in your grammar. For example, if you want to emit an AST with 
nodes that precisely specify what is happening at a particular point in 
your code, you may opt to emit tokens like "FUNC_WITH_NO_ARGS" instead 
of emitting a function and then having to re-parse the argument info out 
of the tree nodes.

That would require putting

tokens { FUNC_WITH_NO_ARGS; }

in your grammar to let Antlr know that you intended to use that symbol 
later.

=Austin


Cristian Peraferrer wrote:
> Hi,
>
> I'm sure it's a newbie's typical question from who never read the 
> manual, but I'm not able to find the answer in the ANTLR's book.
>
> Why I should use the "tokens" section? It's necessary in every 
> grammar? why?
>
> Thanks and apologies for the (stupid?) question.
>
>
> Cristian
>
> -- 
> GPG Key-ID: 0x564903FA - JID: corellian at swissjabber.ch 
> <mailto:corellian at swissjabber.ch>
> ------------------------------------------------------------------------
>
> No virus found in this incoming message.
> Checked by AVG Free Edition. 
> Version: 7.5.488 / Virus Database: 269.14.8/1066 - Release Date: 10/12/2007 11:10 AM
>