[antlr-interest] Rule coverage testing

[Jim Idle]

On 09/21/2009 08:21 AM, Micke Hovmöller wrote:
> There is a code coverade plugin in NetBeans, so I tried with that.
>
> It sort of works. My grammar is 656 lines. The resulting parser is 
> 7074 lines, both including whitespace.
>
> The coverage report says that my tests cover 2338 of 2966 lines. 
> Unfortunately, I can't find out how to break this down by function. 
Sounds like you need to use a different plugin - the coverage tool for 
netbeans says it is from Netbeans beta. There are quite a number of 
tools but Cobertura seems to be most widely used.

> The parser file is color coded per line, but I can't see the 
> difference between lines counted as whitespace and lines that are not 
> covered but really should be.

No options?
>
> Also, a large amount of the uncovered lines are exception handling, 
> most of which is automatically generated, and I'm not interested in 
> trying to come up with test cases for that, if that is even possible 
> in every case.

Yes, and no. Exceptions for parsing you want to cover right? People 
forget that a good regression test for a parser is also to test what it 
does with incorrect input - good and expected message vs "Null pointer 
exception at...."

>
> Finally, some of the stuff doesn't make any sense to me. For instance, 
> towards the end is this code:
> ---
>     static final String DFA56_eotS =
>         "\33\uffff";
...
>     static final String[] DFA56_transitionS = {
>             
> "\1\3\1\uffff\1\2\2\uffff\1\1\1\15\1\22\2\uffff\1\10\1\4\1\16"+
> ( This last string goes on for 40 lines)
> ---
>
> The last static final String line is marked as covered, but not the 
> others. 
As those strings are static and final, then unless the JVM is trying to 
be clever and only initializing them on use (I am pretty sue it does not 
do this, but generates a set of iload istore), then I think that the 
plugin just isn't very good at highlighting. These are the DFAs with run 
length encoding to save on the massive class that would otherwise be 
generated for DFA initialization.

> And it's quite unclear to me if the 40 line string is counted as 1 
> line, 40 lines (all covered) or 1 covered and 39 uncovered lines. I 
> suspect the first, but have found no good documentation and no proper 
> reports.
>
> It's possible (no, make that "likely") that I am doing something 
> wrong, and I would very much appreciate pointers in the right direction.

I don't think you are. I think you just need a better coverage tool.

http://cobertura.sourceforge.net/

Jim