[antlr-interest] Why no links to ANTLR 3.0 on www.antlr.org??

Geir Ove Skjaervik geiroves at online.no
Thu Jun 9 11:42:05 PDT 2005


Hello Terence,

(got your name right this time I hope !)

<<the problem you have though is a general "how do I build an
interpreter" problem, which has nothing to do with ANTLR really.  ANTLR
is just the structure recognizer.  You need to learn about symbol
tables, exec   environments, and a bunch of other stuff.  Perhaps my
grad course  notes can help you out there.>>

Well, I do understand symbol tables et. Al: I designed my own language
& wrote the Parser and Stack machine for it 10 years ago in 6 weeks, and
the current ANTLR version is also running fine. However, I had the
impression that ANTLR was indeed built to build interpreters: Why else
all these Tree Walking examples that adds numbers?

Anway, it is quite possible to build Interpreters with ANTLR, and even
though I see places where ANTLR code can be improved, the current
version I have runs 5 times faster (in Java) than the C++ Stack Machine
version that was written 10 years ago on the **same** computer!

The problems is that after a while technical terms will be forgotten:
Literals, Tokens, String literals etc. It would be nice if ANTLR did a
short **practical** recap: I do know the current docs explains Tokens
and Virtual Tokens, and today when I read it, I think I understand it:
However, I did not the first times I read it!

I do really look forward to your book(s) on ANTLR: If I have the time, I
will be happy to comment on the contents.

Hope my feedback has not been too negatively biased: I was not meant to
be; but I admit to some frustration over getting stuck in the docs :-) 

Finally a resounding THANKS to you for bringing this great tool to the
Parsing Community :-) Hope that you don't forget that ANTLR can indeed
be used to interpret code; That is at least my main goal using ANTLR! 

I will also write a debugger for the same Tree Walker Interpreter, any
hints there?

Regards

Geir Ove

-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Terence Parr
Sent: 9. juni 2005 19:36
To: ANTLR Interest
Subject: Re: [antlr-interest] Why no links to ANTLR 3.0 on
www.antlr.org??



On Jun 9, 2005, at 5:12 AM, Geir Ove Skjaervik wrote:

> Hello Terrnce,

Hi !

> The way you respond to mr. Blomberg tells me that at your level of 
> knowlegde of Parsing and Parser technology you do not any longer 
> comprehend the problems beginners are struggling with: You ask
> questions
> the novice / beginner does NOT know how to answer because (he / she /
> we) are trying to grasp the concept.

Well, I do have to teach students this stuff on a regular  
basis...seems I succeed sometimes...

> Please do take this as constructive feedback as better
> documentation is
> indeed needed to get the beginners going.

I appreciate all feedback!  I want my doc to be as helpful as  
possible.  I plan on an O'Reilly style ANTLR book first then a  
"language translation" book next.

> I have completed my first small Programming Language in ANTLR that 
> exeuctes by Tree Walking,

Good for you!

> and I still do not fully understand what you
> ask Mr. Blomberg... Maybe again it's me...
>
> E.g. You asked:
>
> <<What do you mean by parser  tokens?  Do you mean imaginary tokens 
> inserted by the parser for  which no actual input lexeme exists?>>
> Well,
> why doesn't the docs explain more of this stuff? I still do not

I was actually just wondering what a "parser token" was as *I* hadn't  
heard the term and didn't know what he was asking. ;)

> Any programming tool has its own set of programming Idioms, this is
> also
> the case with ANTLR: When I first started off with ANTLR, there where
> two problems that kept me occupied:
>
> 1) How to parse my language
> 2) How to execute it
>
> So what docs where available to help with this: All over the net there

> are examples on how to execute Expressins ( 2+ 3 etc) using Tree 
> walking. But how to execute an IF statement, or a LOOP ?

True, the examples are more complicated.  The problem you have though  
is a general "how do I build an interpreter" problem, which has  
nothing to do with ANTLR really.  ANTLR is just the structure  
recognizer.  You need to learn about symbol tables, exec  
environments, and a bunch of other stuff.  Perhaps my grad course  
notes can help you out there.

> The more advanced examples like TINY Basic is for a beginner mind 
> boggling. Why not make a minimalistic example that shows BOTH how to 
> parse and execute:
>
> a) Expressions
> b) statements
> c) loops
> d) conditionals
> e) execute simple function calls

Sounds great!  When you finish with yours, can you post it or some of  
it?

> One of the "Patterns" I have found is to create what I call "Holders":

> They are ANTLR Tree Walker subtrees created duing Parsing that do not 
> immediately execute when the tree is walked, but can be called to
> do so
> conditionally by calling another Tree Walker Rule that executes the
> Holders. This is at least how I managed conditionals and loops.
>
> Then the docs need explain how to make the language case sensitive or 
> not (I still haven't got ANTLR to accept my camel-case grammar with 
> keywords like "myKeyword").  Also the concept of
>
> For the Lexer we need to understand how to parse more complex numbers 
> like 1e10 etc. How to define a charVocabulary is also important and
> not
> trivial to the beginner.

Yes, this I hope is much easier in 3.0...

> Then it is the issue of reporting Errors and thus to get hold of
> Source
> Code positions. I may have forgotten some details here, but I think I
> had to implement my own ASTFactory to get hold of the source position
> (getColumn() getLine()) for my Tokens (here we go again: What  
> Tokens?).

Yes, this was a big mistake in 2.x and the tree parsers essentially  
suck at error reporting.

> I also discovered that the concept of Line and Column is NOT very 
> usefull. I will try to explain why: At least Java Editors want the 
> OFFSET in the text to highlight a position (like we want to do to 
> highlight a parsing error).  Now, in my case I wanted to hide parts of

> the source to the user: Call it Headers and Footers of my language,
> so I
> parsed out the part I wanted to show to the user, and wanted to report
> error positions releative to this: It then turns out that the  
> concept of
> Line and Column breaks down, and that indeed OFFSET is the only  
> sensible
> positioning system. So I changed my lexer to only uppdate column and
> never increment line: Thus getColumn() is now the offset!

In 3.0, the column is now char position in line and the token doesn't  
have a string--it has indexes into the text buffer start/stop for  
that token :)

> There are more problems I have been struggling with, but this is
> what I
> remember right now.
>
> Finally, there is the issue of licensing: Can you PLEASE put up a link

> to the licensing policies that exists for version 3.0?

I give my students a demerit when they ask a quesiton that is on the  
first page of google.  In this case, "antlr  license" reveals the  
first entry as your answer:

http://www.antlr.org/license.html

> I know it will be under the BSD license:  Where do I find it?

Try google again and the first entry is what you want.

> And, as reading foreign language license polices is not the simplest 
> thing in the world, can someone in plain english explain whether
> the BSD
> license still gives me the right to use ANTLR 3.x and onwards for
> commercial usel, or am I know stuck with 2.x

BSD says you can do what you want, but don't sue me.  Also, i make no  
guarantee that the code won't destroy all life on this planet. ;)  It  
means you can do anything you want commercially with it. :)

Ter
--
CS Professor & Grad Director, University of San Francisco Creator, ANTLR
Parser Generator, http://www.antlr.org Cofounder, http://www.jguru.com






More information about the antlr-interest mailing list