[antlr-interest] gUnit: Test of a parser rule always fail

[Simon Marchi]

Hmm that's true.

I first tried adding this to remove whitespaces:

WHITESPACE: (SPACE | LF | TAB) {$channel = HIDDEN;};
fragment SPACE: ' ';
fragment LF: '\n';
fragment TAB: '\t';

but that did not fix the problem. I also tried with {Skip();} with the
same results. I then changed my parser rule to a simpler one, having
only one token, no whitespace:
fullName: FIRSTNAME;

and changed the test accordingly:
fullName:
  "SIMON" OK

But it still fails the test.

I tried to do a little bit of debug in the gUnit code, and it appears
that the gUnitExecutor always enters this condition:
/** Invalid input */
if ( tokens.index()!=tokens.size() ) {
    //throw new InvalidInputException();
    ps2.print("Invalid input");
}

In my last example (with fullname: FIRSTNAME;), tokens.index() returns
1 while tokens.size() returns 2. It appears tokens.size() returns the
number of tokens read by the lexer, which is contains the tokens
"SIMON" and "<EOF>". Also, if I understand,correctly, tokens.index()
indicates the index of the next token to be read, which is also the
number of tokens consumed. Comparing both should tell if we consumed
all the tokens present in the input with our grammar rule, is it
correct ?

If you have any tip or explanation that could help me, thank you in advance !

Simon


On Sat, May 21, 2011 at 16:49, Jim Idle <jimi at temporal-wave.com> wrote:
> Your lexer does not cater for spaces.
>
> Jim
>
>> -----Original Message-----
>> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
>> bounces at antlr.org] On Behalf Of Simon Marchi
>> Sent: Friday, May 20, 2011 3:33 PM
>> To: antlr-interest at antlr.org
>> Subject: [antlr-interest] gUnit: Test of a parser rule always fail
>>
>> Hello !
>>
>> I a trying to automatize the testing of my grammar using gUnit, but I
>> ran into the following problem: when I try to test a parser rule, the
>> input string is never recognized. The lexer rules can be tested without
>> problem. I made a simple example to illustrate:
>>
>> Grammar file Cool.g:
>>       grammar Cool;
>>
>>       options {language = Java;}
>>       @lexer::header {package a.b.c;}
>>       @header {package a.b.c;}
>>
>>       // Parser rule
>>       fullName: FIRSTNAME LASTNAME;
>>
>>       // Lexer tokens
>>       FIRSTNAME: 'SIMON';
>>       LASTNAME: 'MARCHI';
>>
>> gUnit file Cool.gunit:
>>       gunit Cool;
>>
>>       @header {package a.b.c;}
>>
>>       // The test case
>>       fullName:
>>         "SIMON MARCHI" OK
>>
>> Normally, the test case should pass, since the input is part of the
>> grammar. I use the ANTLR plugin inside Eclipse, and pasting the input
>> ("SIMON MARCHI") in the interactive interpreter tells me that it is
>> accepted by the rule fullName. However, the test fails. If I change
>> from OK to FAIL, then the test passes (obviously).
>>
>> Is there anything I am doing wrong here ?
>>
>> Thank you,
>>
>> Simon
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
>> email-address
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>