[antlr-interest] Why no links to ANTLR 3.0 on www.antlr.org??

[Geir Ove Skjaervik]

Hello Terrnce,

The way you respond to mr. Blomberg tells me that at your level of
knowlegde of Parsing and Parser technology you do not any longer
comprehend the problems beginners are struggling with: You ask questions
the novice / beginner does NOT know how to answer because (he / she /
we) are trying to grasp the concept.

Please do take this as constructive feedback as better documentation is
indeed needed to get the beginners going.

I have completed my first small Programming Language in ANTLR that
exeuctes by Tree Walking, and I still do not fully understand what you
ask Mr. Blomberg... Maybe again it's me...

E.g. You asked: 

<<What do you mean by parser  tokens?  Do you mean imaginary tokens
inserted by the parser for  which no actual input lexeme exists?>> Well,
why doesn't the docs explain more of this stuff? I still do not
understand this fully, I have simply followed some examples and followed
that pattern without fully understanding what is going on: Hard to admit
maybe, but it's the truth: I have a schedule to try to keep, and I need
to show some progress in the project: Facts of life.



Any programming tool has its own set of programming Idioms, this is also
the case with ANTLR: When I first started off with ANTLR, there where
two problems that kept me occupied:

1) How to parse my language
2) How to execute it

So what docs where available to help with this: All over the net there
are examples on how to execute Expressins ( 2+ 3 etc) using Tree
walking. But how to execute an IF statement, or a LOOP ?

The more advanced examples like TINY Basic is for a beginner mind
boggling. Why not make a minimalistic example that shows BOTH how to
parse and execute:

a) Expressions 
b) statements
c) loops
d) conditionals
e) execute simple function calls

One of the "Patterns" I have found is to create what I call "Holders":
They are ANTLR Tree Walker subtrees created duing Parsing that do not
immediately execute when the tree is walked, but can be called to do so
conditionally by calling another Tree Walker Rule that executes the
Holders. This is at least how I managed conditionals and loops.

Then the docs need explain how to make the language case sensitive or
not (I still haven't got ANTLR to accept my camel-case grammar with
keywords like "myKeyword").  Also the concept of 

For the Lexer we need to understand how to parse more complex numbers
like 1e10 etc. How to define a charVocabulary is also important and not
trivial to the beginner.

Then it is the issue of reporting Errors and thus to get hold of Source
Code positions. I may have forgotten some details here, but I think I
had to implement my own ASTFactory to get hold of the source position
(getColumn() getLine()) for my Tokens (here we go again: What Tokens?).

I also discovered that the concept of Line and Column is NOT very
usefull. I will try to explain why: At least Java Editors want the
OFFSET in the text to highlight a position (like we want to do to
highlight a parsing error).  Now, in my case I wanted to hide parts of
the source to the user: Call it Headers and Footers of my language, so I
parsed out the part I wanted to show to the user, and wanted to report
error positions releative to this: It then turns out that the concept of
Line and Column breaks down, and that indeed OFFSET is the only sensible
positioning system. So I changed my lexer to only uppdate column and
never increment line: Thus getColumn() is now the offset!

There are more problems I have been struggling with, but this is what I
remember right now.

Finally, there is the issue of licensing: Can you PLEASE put up a link
to the licensing policies that exists for version 3.0? 
I know it will be under the BSD license:  Where do I find it? 

And, as reading foreign language license polices is not the simplest
thing in the world, can someone in plain english explain whether the BSD
license still gives me the right to use ANTLR 3.x and onwards for
commercial usel, or am I know stuck with 2.x


Regards 

Geir Ove
Norway

-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Terence Parr
Sent: 5. juni 2005 02:12
To: ANTLR Interest
Subject: Re: [antlr-interest] Why no links to ANTLR 3.0 on
www.antlr.org??


On Jun 4, 2005, at 1:47 AM, Gerald B. Rosenberg wrote:
> I will try to take a constructive stab at this.  Given that the 
> problem space and tool are respectively very complex and capable of
> very complex operation, the documentation is just too eager.

I assume you meant "meager" ;)

>   It presents something basic and then jets off mixing a bunch of 
> fairly complex potential uses together which makes following and
> understanding any one very difficult.

Damn!

> For example, under http://www.antlr.org/doc/trees.html#_bb4, there is 
> brief introduction to marking lexer tokens as roots.  Ok.  Then
> some stuff about building trees.  Nothing there about parser tokens  
> as roots.

Apparently the doc is worst than that!  What do you mean by parser  
tokens?  Do you mean imaginary tokens inserted by the parser for  
which no actual input lexeme exists?

>   I think that is what I need for my problem -- I need to understand 
> better how ANTLR wants to mark and construct parser
> trees.  So, is marking a new parser token as a root even possible?   
> Well, keep looking down.  Token translation.  Ok.  Token  
> factories.  Ok, I guess.  Heterogeneous ASTs.  Ok, real advanced.   
> Fairly certain that is not applicable to my initial needs.

Interesting...define parser token for me and I'll try to help.

> Finally! An first example ("An Expression Tree Example").  Maybe that 
> will walk me through enough to understand how ANTLR wants  me
> to solve my problem.
>
> Problem statement: a simple value accumulator implemented using a 
> simple tree.  Ok, not really close to my case I think, but sounds
> like it has a similar level of complexity so I should be able to  
> learn bunches from it.
>
> OH, NO: Third paragraph, the "formal statement of the explanation of 
> the solution to follow":
>
> "The AST operator nodes must combine the results of computing the 
> value of their two subtrees.  They must perform a depth-first walk
> of the tree below them.  For fun and to make the operations more  
> obvious, the operator nodes define left() and right() instead,  
> making them appear even more different than the normal child- 
> sibling tree representation.  Consequently, these expression trees  
> can be treated as both homogeneous child-sibling trees and  
> heterogeneous expression trees."
>
>
> ??? Fun and obvious???  Make them appear more different???  A
> straight first example?  No.  More accurately an advanced  
> transvestite example.

Did any of the examples or tutorials help?  The doc is not  
particularly friendly.

> There are so many issues working in this example, I challenge
> anyone not already an expert to readily tell what design features  
> pertain to what -- what is there for fun and what select parts are  
> essential for the particular purposes relevant to a relative new- 
> comer's basic issues of understanding.  The question is, would  
> someone not capable of authoring this example in the first place,  
> now understand matters well enough to go back and author the  
> simple, straight-forward version?

Sounds like we have a navigation issue not doc issue (though I plan  
on fixing the doc).  Did you see my course notes and all that jazz?

> Now, don't get me wrong: I am not advocating plodding
> documentation.  Enthusiasm is a great tool for maintaining interest  
> in the narrative, particularly on paper.  There just needs to be  
> something of a middle ground (at least for us middle of the road  
> programmers).

Can you lay out precisely what you've read / looked at what you  
haven't?  That will tell me a lot!  As I say, it may be simply that  
the information is not available in the proper order or even defined  
in what order you should look.

Ter