[antlr-interest] Simple Grammar breaks ANTLRWorks Interpreter & Debugger?

[David-Sarah Hopwood]

Gavin Lambert wrote:
> At 10:12 14/08/2009, consiliens at gmail.com wrote:
>  >QuizLexer lexer = new QuizLexer(new ANTLRFileStream(input));
>  >CommonTokenStream tokens = new CommonTokenStream(lexer);
>  >// prints 0
>  >System.out.println(tokens.size());
>  >
>  >So according to this there are no tokens, which would explain 
>  >the blank debugger Input window. If the ANTLRWorks Interpreter runs
>  >the code correctly, why not the Debugger or unit tests?
> 
> Actually that's a bit deceptive :)
> 
> The stream starts out empty, and thus .size() returns 0.

You're right, and that seems like a bug. In CommonTokenStream:

00315     public int size() {
00316         return tokens.size();
00317     }

should be
          public int size() {
              if ( p == -1 ) {
                  fillBuffer();
              }
              return tokens.size();
          }

Probably consume() also needs the same fix. (There are other methods
that don't check for p == -1, but they're declared as protected.)

> Generally, I find the debugger significantly more reliable than 
> the interpreter ever is...

Same here. The interpreter can't work properly on grammars that have any
significant reliance on target-language code.

-- 
David-Sarah Hopwood  ⚥  http://davidsarah.livejournal.com