[antlr-interest] No errors on missing tokens?

[Kirby Bohling]

Naturally I forgot to add ANTLR Interest to by Reply-To, so the
information is captured in case it is useful to others.   I didn't
think you'd mind my inclusion of your reply also.

On Tue, Jun 28, 2011 at 12:52 PM, Bill Andersen <andersen at highfleet.com> wrote:
> Thanks, Kirby
>
> Yup.  I triple (no.. quadruple) checked that.  I cleaned the project, manually deleted the .tokens files, etc.  All the standard stuff.  Sorry...  My questions to the list tend to be a little ADD-like and I leave out important details like that.
>
> This is a genuine bug unless I'm missing something really obvious.
>
> BTW, great reply.  Doesn't help me but it will help ANTLR newbies.
>
>        .bill
>
> On Jun 28, 2011, at 13:45 , Kirby Bohling wrote:
>
>> On Tue, Jun 28, 2011 at 12:34 PM, Bill Andersen <andersen at highfleet.com> wrote:
>>> Partial answer to my own question, in case anyone is interested.
>>>
>>> I added all the missing token definitions to my lexer.  When I then compiled the lexer and parser and looked at the .tokens files, all the tokens and their type #s lined up.  So it seems that:
>>>
>>> 1) Given a complete lexer, the compilation of the parser grammar is using the lexer's .tokens properly
>>>
>>> 2) Compiling the parser yielded no errors for a complete lexer .tokens file
>>>
>>> 3) Compiling the parser signalled no errors for an incomplete lexer .tokens file
>>>
>>>
>>> On Jun 27, 2011, at 22:21 , Bill Andersen wrote:
>>>
>>>> Hi Folks
>>>>
>>>> I have a grammar split into a lexer grammar and parser grammar
>>>>
>>>> I have not yet defined all the token types in the lexer, so the parser grammar is making reference to tokens that are not in the lexer .tokens file.  The lexer .tokens file has a max token # of 108.  This should produce errors but is not.  BTW, compiling the parser grammar yields a .tokens file with a max token # of 158.
>>>>
>>>> Can anyone tell me what's going on here?  Thanks
>>>>
>>>>       .bill
>>>>
>>
>> Looking at your original question, did you have the correct
>> declaration tokenVocab in your options section in the Parser/Tree
>> grammars?
>>
>> options {
>> ...
>>    tokenVocab=MyNameHereLexer;
>> ...
>> }
>>
>> I use Maven, so it always generates both the Lexer and Parser at the
>> same time (and if you have TreeGrammars, you'll definitely need this
>> there).  I'm not sure if you need to process both in a single run or
>> not.  I always declare all my virtual tokens in my Lexer so I have one
>> file that has them all, but I know that it isn't necessary (I put them
>> temporarily in the Parser while until I get everything working then
>> migrate them to the lexer as I add rules that need virtual tokens).
>> However, if you don't declare the tokenVocab and you have split
>> Lexer/Parser grammars it definitely causes problems with tokens not
>> being in sync, and the types of things you are describing.
>>
>> Not sure if this is your problem, but it is how I have problems like
>> what you are describing.  So even if it doesn't help you, hopefully it
>> helps others quickly spot one of my beginner mistakes with ANTLR.
>>
>> Kirby
>
> Bill Andersen
> Highfleet, Inc. (www.highfleet.com)
> 3600 O'Donnell Street, Suite 600
> Baltimore, MD 21224
> Office: +1.410.675.1201
> Cell: +1.443.858.6444
> Fax: +1.410.675.1204
>
>
>
>
>
>