[antlr-interest] A postmortem of my use of antler

[Benjamin Shropshire]

Kay Röpke wrote:
> Hi!
>
> On Mar 10, 2008, at 9:29 PM, Benjamin Shropshire wrote:
>> The concept of ANTLR is a really good idea.
>>
>> The execution is fairly good (It could be better in a few places but 
>> it's not worth complaining about.
>
> Could you expand on the places you feel are sub-standard? It would be 
> good to know so we can improve that, if possible.
once in a while a syntax error in the input will throw an exception, and 
some of the errors (parser, semantic and runtime) are nearly useless 
unless you known what's going on, but that's about it.

>> In all that, it took me over 2 weeks to even notice that the expected 
>> mode of operation is to lex, AST, work with AST. I saw mentions of 
>> AST and assumed that this was antlr's turm for the type of output 
>> that yacc and bison typically are sued to generate. This is just one 
>> example of a cases where the antlr docs seemed to assume a 
>> understanding of things that I didn't have. And I'm not talking 
>> general ignorance. I'm referring to ignorance of the specifics of 
>> antlr. The best I can do to describe my frustration is this: "the 
>> antlr doc's are not good enough to let a new user figure out what 
>> questions to ask." I'm not so dumb that I can't figure antlr out 
>> (theres nothing complicated with any part of it that I've used) I 
>> just wasn't asking the right questions and had no way to find that out.
>
> What kinds questions are these? What cannot be figured out by 
> reading/searching the ANTLR wiki and/or posting to this mailing list? 
> The archives generally turn up pretty good explanations, too, and are 
> searchable via google.

"how is antler /expected/ to be used?" is the big one. That once should 
be on page one. I have yet to find a direct answerer anywhere.

> But, with all due respect, the "expected mode" of a parser generator 
> like ANTLR is to lex, parse, and do something with the resulting 
> tree(s). Any kind of reading about the subject matter introduces the 
> concept. Even for someone not having had the benefit of classes in 
> university the wikipedia pages should provide enough information to 
> figure that out. 2 weeks sounds excessive and exaggerated to me.

The important destruction is that in antler you let antler build the AST 
the first time though. swapping the order of chapter 3 in the book would 
make this clearer (First, how to build an AST from the input, then how 
to process that, then how to avoid the AST all together) Also that info 
should be /explicitly stated/ near the top of section 1.1

>> Now I will confess that I have yet to read the manual cover to cover, 
>> but I think I've hit all of the intro material in it and in the web 
>> pages. If that material doesn't give the user something to hold onto, 
>> I fear that many users a just going to give up on antlr after only a 
>> few minutes to an hour. In retrospect that is what I /should/ have 
>> done. My bosses opinion is that antlr is not giving us much enough 
>> value over writing the parser by hand. I'm a bit embarrassed to say, 
>> I think he's correct and I could have/should have made that call a 
>> lot sooner.
>
> I hereby question the boss' and colleague's opinions. For a very 
> simple, one-time recognizers there might be truth in it, but if you 
> expect to have to maintain the recognizers in the future, it's madness 
> to not use some kind of generator. After all, writing correct 
> recognizers is tricky, especially if the language isn't trivial.

This is based on actually trying it, from a LOC standpoint, it's a toss 
up and the hand done code is a lot easier to debug. At this point if I 
do have to go with a generator I would plan on using flex/bison and 
linking that into the C# (as bad as that is)

>> Now for some rhetorical questions:
>>
>> --What is ANTLR? Is it an academic research project? A source of work 
>> for grad students? A FOSS utility intended for the general community? 
>> A toy?
>
> Warning: Rhetorical answers ahead!
> It's
>  a) A research project.
>  b) Provides grad students with a tool they can use to learn about 
> parsing.
>  c) Provides grad students with a tool they can use to do more 
> advanced work and not spend the entire class on implementing 
> recognizers, so they can move on to implement 
> languages/compilers/whatever once they grokked parsing enough to 
> understand the value of _not_ writing recognizers by hand.
>  d) A FOSS utility intended for the general community, lowering the 
> bar to write recognizers. Ever used and maintained a 15000 line yacc 
> grammar?
>  e) A high-quality tool widely used in the industry and open source 
> community.

In that order?

>>
>> --Who is the intended user base? It's own developers? 
>> Language/grammar theory people? Professional programming gurus? The 
>> programming world at large?
>
> Yes, yes, yes, yes ;)

In what order?

>> I don't need answers to those, I'd be interested but it's academic at 
>> this point. If ANTLR is supposed to be what I think it's supposed to 
>> be, I think that ANTLR would benefit by the consideration of those 
>> questions.
>
> I agree there's a need to lower the bar of entry even further, but I 
> think no tool has lowered it to the level ANTLR has. At least not yet.

I have a parser generator that I wrote that is easier to understand 
(manly because it is much simpler and consequently less expressive). The 
extraneous, as in not strictly needed to build parsers (but useful), 
parts are what gets in the way.

>
> I don't think anything like that will happen, either. To use tools you 
> need at least a basic understanding of the concepts. Don't need to be 
> an expert, but understanding helps a lot. If someone does not 
> understand the basics of parsing, it's kinda hard to write parsers, 
> with tools or without.

I understand parsing. I didn't understand what antlr expected me to do 
with the parser.

>> p.s. I might get to use ANTLR on another project (only, oddly enough, 
>> because the default rendering of AST looks like lisp)
>
> Yeah, it's odd ;)

We will need to use lisp to process C code. Thus...

>> p.p.s. Another guy on that last project (with a much different 
>> personality than me) also took a look at antlr and more or less 
>> agreed with me on the deficiencies of the antlr docs.
>
> What is the background of those people? Is it just the docs? Is it the 
> tool?

CS from the hardware side.
Mostly the docs. The tool works fine for the most part and the design is 
not bad to work with after you figure it out.