[antlr-interest] Preserving Original Token Sequence In ASTs

[Terence Parr]

Guys,

Had a chat with Greg Benson (an OS maniac and fellow professor here at  
USF) yesterday and he described some transformations he had done in  
JavaCC to build efficient parallel processing libraries (special  
encodings for various types etc...).  He had a nice translator solution  
where it was easy to get the original token order from his trees so he  
could do easy Java->Java transformations.  We talked about the  
difference between parse trees and ANTLR's ASTs and how ASTs are what  
you want for translation, but it's hard to get the original token  
sequence back out.  I decided to tackle the issue today and have posted  
an article describing my results.  Thanks for the motivation, Greg!  We  
may have the best of both worlds now. :)

Here is what I conjured up and just posted to antlr.org:

http://www.antlr.org/article/preserving.token.order/ 
preserving.token.order.tml

"You want ASTs instead of parse trees for translation, but ASTs have  
the token order shuffled from the one-dimensional input sequence into a  
two-dimensional tree structure.  Consequently, it is often hard to spit  
the original token sequence back out.  You are required to build a tree  
walker that spits text back out.  This article describes an  
experimental means of supporting easy x to x translation without  
modifying ANTLR itself.  I provide modified token, token stream, tree  
node, and AST factory objects along with a simple grammar to  
demonstrate their use.  This is damn cool stuff and will be standard  
equipment on ANTLR 3."

Let me know what you think. :)  The solution is cute if ya ask me. ;)

L8R G8Rs,
Terence
--
Professor Comp. Sci., University of San Francisco
Creator, ANTLR Parser Generator, http://www.antlr.org
Co-founder, http://www.jguru.com
Co-founder, http://www.knowspam.net enjoy email again!
Co-founder, http://www.peerscope.com pure link sharing




 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/