[antlr-interest] Simple Grammar breaks ANTLRWorks Interpreter & Debugger?

[David-Sarah Hopwood]

Jim Idle wrote:
> David-Sarah Hopwood wrote:
>> Jim Idle wrote:
>>> David-Sarah Hopwood wrote:
>>>> Gavin Lambert wrote:
>>>>       
>>>>> At 10:12 14/08/2009, consiliens at gmail.com wrote:
>>>>>  >QuizLexer lexer = new QuizLexer(new ANTLRFileStream(input));
>>>>>  >CommonTokenStream tokens = new CommonTokenStream(lexer);
>>>>>  >// prints 0
>>>>>  >System.out.println(tokens.size());
>>>>>  >
>>>>>  >So according to this there are no tokens, which would explain 
>>>>>  >the blank debugger Input window. If the ANTLRWorks Interpreter runs
>>>>>  >the code correctly, why not the Debugger or unit tests?
>>>>>
>>>>> Actually that's a bit deceptive :)
>>>>>
>>>>> The stream starts out empty, and thus .size() returns 0.
>>>>>         
>>>> You're right, and that seems like a bug. In CommonTokenStream:  
>>>    
>>> No - you need to calls something like LT(1) first - otherwise all methods have 
>>> to check for init which is an overhead that isn't really needed.
>>
>> Except that all public methods except .size() and .consume() already do
>> (directly or indirectly) perform this check. That is, almost all of the
>> overhead is being paid already -- and is needed, since the generated
>> parser code doesn't ensure that it calls LT, or any other single method,
>> first.
> 
> Parser calls token stream, it checks for empty stream, then fills buffer.

Yes, and that check is implemented by testing for p == -1 in all of the
public methods of CommonTokenStream except .size() and .consume().

The omission of .size() and .consume() is a bug. Adding the check to
.size() wouldn't have any significant performance impact, since it's
not a frequently called method. Not adding it to .consume() on performance
grounds would be premature optimization. (There are obvious optimizations
to .consume() and .skipOffTokenChannels(int) that would more than offset
this, *if* profiling showed that it were important to do so.)

-- 
David-Sarah Hopwood  ⚥  http://davidsarah.livejournal.com