[antlr-interest] Re: woohoo! ParseTrees for free, Debugging for me!

[Terence Parr]

On Sunday, November 30, 2003, at 12:16  AM, lgcraymer wrote:

> --- In antlr-interest at yahoogroups.com, Terence Parr <parrt at c...>
> wrote:
>>
>> On Saturday, November 29, 2003, at 10:16  PM, lgcraymer wrote:
>>
>>> Ter--
>>>
>>> If I interpret this right, you're actually visualizing parse
> trees
>>> without generating them.
>>
>> Nope, i build parse trees and then rewalk them to get step i of
>> derivation.  You can ask for any intermediate step or all of 'em :)
>
> That is impressive, then--you've discovered a whole new approach to
> instrumenting ANTLR.  I assume that you've made the overrides user
> programmable?

Well, for now it's just an "article" on the website with some sample 
code...later we'd have to allow people to specify their own parse tree 
nodes instead of my simple ones...  As for user programmable, for now, 
you can turn on or off by adding the overrides and turning on/off 
-traceParser.

>   It sounds like an approach that might support an
> ANTLR test harness for grammars.

Heh yeah!  That's a great idea, dude!  Sweet!  This way we can check to 
see if all derivation steps are identical.  Previously, testing parsers 
was really hard as it says yes or no depending on errors; that's all.   
  Answering "yes" is not much of a test as an empty main program gives 
the same answer. ;)  Yeah!  This is the testing harness I've been 
afraid would be extremely difficult!  We've got it, by jove!

The other beauty of my debugging parse tree / derivations is that it 
does not affect code gen in any way, meaning that normal tree 
construction occurs as before.  You can create both parse tree and 
regular AST :)  I'm going to verify that today.

All you have to do is say

class T extends Parser(ParseTreeDebugParser);

rules...

and then turn on -traceParser.  Your parser will answer getParseTree() 
and getLeftmostDerivation(step-number) and getLeftmostDerivation.

The derivation thing is really cool...had to use my brain for about an 
hour to realize that I'm essentially counting the number of subtree 
roots as I do a depth-first to simulate replacements in the derivation 
steps.

The code is very simple and will be posted this morning :)

Man, looking at the derivation tree makes me wonder why I didn't do 
this before.  It makes it completely obvious how input is matched...wow!

Also, the derivation steps begs the question: can we make a simple XSLT 
equivalent that uses a normal human syntax like

replace "if ( <expr> )" with "if ( (int)<expr> )";

or whatever?  Hmm....hell even a global string replace would work on 
each derivation step and then let derivation proceed as normal.  
Interesting...

>>> Modifying ANTLR to build parse trees is even easier--add the root
>>> node creation code in genRule (XXXCodeGenerator) and modify
> refRule
>>> in DefineGrammarSymbols.java, then adding an option recognizer to
>>> ParserGrammar.java.  About 10-15 lines total, unless I've missed
>>> something.
>>
>> Sounds great!  I just wanted something quickly that didn't require
>> modifications to ANTLR.  Figured I'd break something ;) and we
> will
>> have a bug release first before features. :)
>
> It's not quite that easy to break ANTLR internals--you have to look
> at them cross-eyed ... :).

Well, w/o regression tests, I don't trust myself ;)  The ANTLR 3 stuff, 
as you know since you've looked at it, has plenty-o-tests. :)  I'm a 
testing fool nowadays instead of the old academic prototype guy I was. 
:)

Ter
--
Professor Comp. Sci., University of San Francisco
Creator, ANTLR Parser Generator, http://www.antlr.org
Co-founder, http://www.jguru.com
Co-founder, http://www.knowspam.net enjoy email again!
Co-founder, http://www.peerscope.com link sharing, pure-n-simple




 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/