[antlr-interest] unit tests

[Terence Parr]

Hi Jeremy,

Very interesting work! at a recent GPCE2006 conference, Martin  
Bravenboer showed me his interesting tools for testing GLR-based  
Stratego stuff.   After that, I decided to build GUnit using the same  
ideas Martin had.  You can do each thing things without having to  
resort to Java code:

expr:
	"3+4" -> ^(+ 3 4)
	"3+4*5" -> ^(+ 3 ^(* 4 5))

etc...

  where you could test output, parse tree, AST, etc. Naturally, this  
could simply generate JUnit code.

Ter


On Dec 9, 2006, at 10:02 AM, Jeremy D. Frens wrote:
> Sorry to jump into the discussion late.  It's too easy to let things
> slide while on a sabbatical...
>
> Robert Hill wrote:
>> Hi all, just want to open up a bit of a discussion, how does  
>> everyone else
>> handle unit testing for a compiler/code generator.
>> Or if you're using JUnit/NUnit how do test your Antlr output ?
>>
>> File compares on the output sources? How do you test individual  
>> constructs?
>> Hundred of files( or embedded strings) with each one having a  
>> single example
>> to the construct in question?
>>
>> Just looking for some good ideas ( or examples :) ) on  
>> implementation (I'll
>> be using JUinit btw)
>
> I believe very strongly in unit testing, so while I was writing an
> interpreter using ANTLR (http://nolatte.sourceforge.net), I also  
> built a
> unit-testing library for ANTLR grammars using JUnit.  My results  
> can be
> found on SourceForge: http://antlr-testing.sourceforge.net/ .
>
> Unfortunately, there are a few problems with the ANTLR Testing  
> library:
> first, the website sucks.  Second, the library works (as far as I  
> know)
> only with ANTLR 2.7.6, and it is unaware of JUnit 4.x.  Third,
> ironically, the library is not well tested.  Fourth, I've learned a  
> lot
> about unit testing since I wrote the library, and the overall  
> design of
> the library could stand some reworking.
>
> As an example of the fourth problem, in order to test a tree  
> parser, you
> end up testing the lexer and the parser as well.  I don't know that  
> I'll
> necessarily drop this because it *is* good functional testing.  But a
> unit test should just test *one* action (e.g., one tree  
> transformation)
> in *one* module (e.g., the tree builder).
>
> That's the bad news; here's the good news.
>
> The library is very usable now, even with my warnings.  I've used  
> it for
> my own interpreter; I've used it two or three times in a compiler  
> course
> (at Calvin College, http://cs.calvin.edu/); I've used it twice in a
> Programming Languages course (also at Calvin).
>
> Better yet, the purpose of my sabbatical next semester is to write  
> a lab
> manual (maybe textbook) for a Programming Languages course, doing
> incremental, test-driven development of interpreters (plural!) and  
> other
> language transformations (like a compiler).  It's becoming clear to me
> this semester that I need a better ANTLR Testing library to make this
> lab manual effective.
>
> Some changes that I've planned (not in any real order): switch to  
> JUnit
> 4.1 (or whatever's current); rework the existing library for ANTLR
> 2.7.6; build a new library for ANTLR 3.0; unit test the library;  
> write a
> better website.
>
> I'm very interested in what people think about my ANTLR Testing  
> library
> and what features you'd like to see in such a library.
>
> jdf