[antlr-interest] Possible Antlr defect?

[Terence Parr]

On Jun 9, 2007, at 6:02 PM, Stuart Dootson wrote:

> On 10/06/07, Terence Parr <parrt at cs.usfca.edu> wrote:
>>
>> On Jun 9, 2007, at 5:25 PM, Stuart Dootson wrote:
>> > But - don't all tokens need to be defined *somewhere*? For  
>> example, on
>> > page 179 of the PDF of the book, we find the follwoing statement:
>> >
>> > 'The imaginary token must be defined elsewhere in a grammar or  
>> in the
>> > tokens section.'
>> >
>> > I thought that was the idea of the 'tokens' section - to provide a
>> > mechanism for defining tokens not produced by the lexer, meaning  
>> that
>> > all tokens were defined somewhere? And if all tokens are  
>> defined, you
>> > have a complete list of them that you can validate token usage
>> > against?
>>
>> That "elsewhere in a grammar" is the key.  Just referencing a token
>> defines it as well as using the tokens section.
>> Ter
>
> One final thing that's puzzling me - when I use the same grammar in a
> combined lexer/parser grammar, Antlr does complain, like so (same
> parser grammar as before, same lexer and token types as before):
>
> warning(105): Test.g:14:5: no lexer rule corresponding to token: ID
>
> Is a combined grammar a special case in this respect?

Yup. If you use a token for which there is no lexer, it knows about  
it because it sees the entire set of lexer rules. :) In this special  
case, it knows what the errors are.

Ter