[antlr-interest] Possible Antlr defect?

[Stuart Dootson]

On 10/06/07, Terence Parr <parrt at cs.usfca.edu> wrote:
> Hi. Unfortunately, this is not exactly a bug, just undesirable.  You
> can *add*token definitions to the ones that have already been
> imported.  There are many imaginary tokens you need when doing tree
> construction.  If ANTLR were to warn you upon every new imaginary
> token, it would be a drag.  Perhaps we need an option to warn you of
> these things, or better yet, something in ANTLRWorks that would warn
> you. :)
>
> Ter
>

But - don't all tokens need to be defined *somewhere*? For example, on
page 179 of the PDF of the book, we find the follwoing statement:

'The imaginary token must be defined elsewhere in a grammar or in the
tokens section.'

I thought that was the idea of the 'tokens' section - to provide a
mechanism for defining tokens not produced by the lexer, meaning that
all tokens were defined somewhere? And if all tokens are defined, you
have a complete list of them that you can validate token usage
against?

Stuart