[antlr-interest] Simple Grammar breaks ANTLRWorks Interpreter & Debugger?
David-Sarah Hopwood
david-sarah at jacaranda.org
Sat Aug 15 09:12:10 PDT 2009
Jim Idle wrote:
> David-Sarah Hopwood wrote:
>> Jim Idle wrote:
>>> David-Sarah Hopwood wrote:
>>>> Gavin Lambert wrote:
>>>>
>>>>> At 10:12 14/08/2009, consiliens at gmail.com wrote:
>>>>> >QuizLexer lexer = new QuizLexer(new ANTLRFileStream(input));
>>>>> >CommonTokenStream tokens = new CommonTokenStream(lexer);
>>>>> >// prints 0
>>>>> >System.out.println(tokens.size());
>>>>> >
>>>>> >So according to this there are no tokens, which would explain
>>>>> >the blank debugger Input window. If the ANTLRWorks Interpreter runs
>>>>> >the code correctly, why not the Debugger or unit tests?
>>>>>
>>>>> Actually that's a bit deceptive :)
>>>>>
>>>>> The stream starts out empty, and thus .size() returns 0.
>>>>>
>>>> You're right, and that seems like a bug. In CommonTokenStream:
>>>
>>> No - you need to calls something like LT(1) first - otherwise all methods have
>>> to check for init which is an overhead that isn't really needed.
>>
>> Except that all public methods except .size() and .consume() already do
>> (directly or indirectly) perform this check. That is, almost all of the
>> overhead is being paid already -- and is needed, since the generated
>> parser code doesn't ensure that it calls LT, or any other single method,
>> first.
>
> Parser calls token stream, it checks for empty stream, then fills buffer.
Yes, and that check is implemented by testing for p == -1 in all of the
public methods of CommonTokenStream except .size() and .consume().
The omission of .size() and .consume() is a bug. Adding the check to
.size() wouldn't have any significant performance impact, since it's
not a frequently called method. Not adding it to .consume() on performance
grounds would be premature optimization. (There are obvious optimizations
to .consume() and .skipOffTokenChannels(int) that would more than offset
this, *if* profiling showed that it were important to do so.)
--
David-Sarah Hopwood ⚥ http://davidsarah.livejournal.com
More information about the antlr-interest
mailing list