[antlr-interest] Compiler design workflow - any suggestions?

[Monty Zukowski]

On May 4, 2004, at 12:00 PM, c3i_smc wrote:

> A question from a beginner at this game.
>
> I wondered how the more experienced compiler writers manage the
> process of creating ANTLR code, generating the parser, testing the
> (new) stuff as  an ongoing process.

My GCC grammar uses a simple test methodology.  I have a bunch of files 
named test01.c, test02.c, etc.  My framework will lex, parse & build 
the tree, tree parse, and emit the result back to text.  Then I run the 
original and result through the lexer again to compensate for 
formatting differences and diff those.  Errors show up either as 
exceptions in any phase or if diff returns true.

This is a bit broader than a unit testing framework, but grammar rules 
aren't really independent units like methods.  For instance you can't 
just call into any rule with a fragment of input, you have to call a 
top-level rule so that ANTLR has calculated the lookahead correctly. 
Rules depend on other rules quite a bit, and I find it easier to reason 
about complete input programs that exercise a rule.  Even in C it's 
pretty easy to narrow them down to a specific rule.

The other day I realized that I could write a program to run my test 
files through the program built with -trace turned on and collect which 
rules were hit by which program into a database.  Then I know when I'm 
working on a particular rule I should run those particular tests for 
sure.

Ter's idea of using the parse tree has some merit but I would find it 
cumbersome.  Building trees usually requires splitting rules apart 
which would make you change your unit tests for that rule.  Actually 
any refactoring could change the parse tree.  When I'm developing a 
grammar I don't necessarily know where I'm going.  I know how it should 
act on a complete program but not at the rule level, so that's how I 
like to test.

Also I highly recommend using "local" source control and checking in 
with _every_ build.  It's easy to go astray with grammars and not know 
how far back you need to go to be at a known good state.  I use rcs 
commands in a makefile like this:

rcsfiles = zRCS/select.g,v zRCS/selectLexer.g,v zRCS/Main.cpp,v
rcs : $(rcsfiles)

zRCS/Main.cpp,v : Main.cpp
	ci -l $@
...

"make rcs" will check in any changed files, and check them out locked 
again.  The log messages I enter let me know what I was thinking at the 
time when I was changing stuff.

Monty Zukowski

ANTLR & Java Consultant -- http://www.codetransform.com
ANSI C/GCC transformation toolkit -- 
http://www.codetransform.com/gcc.html
Embrace the Decay -- http://www.codetransform.com/EmbraceDecay.html



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/