[antlr-interest] Testing lexer grammars with gunit

Leon Su leon.antlr at mac.com
Wed Nov 25 00:16:16 PST 2009

On Nov 24, 2009, at 12:17 AM, Gavin Lambert wrote:

> At 20:54 24/11/2009, Leon Su wrote:
>> gUnit P;
>> lexical-rule-name:
>> "input" OK
>> ...
>> By the way, the next release of gUnit will allow you to test a  
>> lexer grammar individually with the syntax: gUnit lexer L;
> Does gUnit only support that kind of limited testing?  (I ask out of  
> ignorance; I've never really looked at it.)
> For lexer rules in particular, "OK" is a fairly meaningless test.   
> What'd be better is something like:
> gUnit P;
>  "abc" ID
>  "abc123" ID
>  "123" INT
>  "a+b" ID["a"] PLUS ID["b"]
>  "a- -b" ID MINUS MINUS ID
> etc.  Then you could do lexer-only testing for lexer grammars and  
> lexer-and-parser testing for combined grammars.

gUnit treats every rule as a unit which is the smallest testable part  
of a grammar, and it tests whether individual units of the grammar are  
fit for use.
Therefore, the test you recommended above could be rewritten in the  
gUnit format as below:

gUnit P;
"abc" OK
"abc123" OK
"a" OK
"b" OK
"123" OK
"+" OK
"-" FAIL

But I also like your idea of the token-stream style testing for lexer  


More information about the antlr-interest mailing list