[antlr-interest] Testing lexer grammars with gunit

Leon Su leon.antlr at mac.com
Wed Nov 25 00:16:16 PST 2009


On Nov 24, 2009, at 12:17 AM, Gavin Lambert wrote:

> At 20:54 24/11/2009, Leon Su wrote:
>> gUnit P;
>> lexical-rule-name:
>> "input" OK
>> ...
>>
>> By the way, the next release of gUnit will allow you to test a  
>> lexer grammar individually with the syntax: gUnit lexer L;
>
> Does gUnit only support that kind of limited testing?  (I ask out of  
> ignorance; I've never really looked at it.)
>
> For lexer rules in particular, "OK" is a fairly meaningless test.   
> What'd be better is something like:
>
> gUnit P;
> LEXER:
>  "abc" ID
>  "abc123" ID
>  "123" INT
>  "a+b" ID["a"] PLUS ID["b"]
>  "a--b" ID DECREMENT ID
>  "a- -b" ID MINUS MINUS ID
>
> etc.  Then you could do lexer-only testing for lexer grammars and  
> lexer-and-parser testing for combined grammars.


gUnit treats every rule as a unit which is the smallest testable part  
of a grammar, and it tests whether individual units of the grammar are  
fit for use.
Therefore, the test you recommended above could be rewritten in the  
gUnit format as below:

gUnit P;
ID:
"abc" OK
"abc123" OK
"a" OK
"b" OK
INT:
"123" OK
PLUS:
"+" OK
"-" FAIL
...

But I also like your idea of the token-stream style testing for lexer  
grammars.

-L


More information about the antlr-interest mailing list