[antlr-interest] philosophy about translation

[Anthony W. Youngman]

In message <454A1879.6040802 at jazillian.com>, Andy Tripp 
<antlr at jazillian.com> writes
>
>>>
>>> Every time I've heard anyone say they NEED MI, operator overloading, 
>>>or Aspects, I've thought
>>> that they really don't need them, they're just not good enough 
>>>programmers to see a clean way
>>> to do without them. I've been programming for 25 years without ever 
>>>NEEDING any of these,
>>> and so have most other programmers.
>>>
>> I programmed for many years without needing pointers. FORTRAN didn't 
>>have them, so I tended not to use them in C. I notice you said *MOST* 
>>other programmers. You're jumping to conclusions - because "all the 
>>programmers I know are too stupid to live without these features" you 
>>conclude that these features "mostly aren't needed". So you'd like to 
>>see them removed from any tool you use ... :-(
>
>I conclude that these features "mostly aren't needed" (really more like 
>"almost never" or even "never") from personal
>experience, not from something you or I said.
>
I thought that was what I said - your personal experience leads you to 
believe they're not necessary ... (and you ignore other people's 
experiences to the contrary :-)
>>>>
>>>>>
>>>> So - you're quite happy to see Antlr crippled to suit you, ignoring 
>>>>the NEEDS of those who need its power!
>>>
>>>
>>> Don't worry, I can't cripple ANTLR.  But yes, I'd be happy to see 
>>>ANTLR4 become a "crippled" version,
>>> in the same sense that Java is a "crippled" version of C++.
>>>
>> And as I said, you're quite happy to ruin the tool for all those 
>>people who need that power ...
>
>Yes, if you consider Java to be "ruining" C++. New tool without all the 
>warts. Those
>who "need the power" of the warts can keep using the old tool.
>
Except you seem to be saying that C++ is unnecessary because Java can do 
everything that C++ is used for.
>>
>>>>>>
>>>>>>
>>>>> But required knowledge of the tool's internals limits the 
>>>>>"average" user's productivity.
>>>>> If I had done AST-based translation, I'd be spending way to much 
>>>>>time worrying about the details
>>>>> of the AST, rather than the syntax and semantics of the two 
>>>>>languages. I demand to spend 95% of
>>>>> my time on *what* to do, rather than *how* to do it. With ASTs, I 
>>>>>found myself spending
>>>>> 95% of my time on *how*.
>>>>>
>>>> In other words, as you said earlier, ASTs are the wrong tool for 
>>>>you. So you seem happy to delete ASTs from Antlr because *you* don't 
>>>>need them, irregardless of what other people *NEED*.
>>>
>>>
>>> Yes, just as I railed against operator overloading and MI in C++, 
>>>and found bliss in Java.
>>> I don't much care that other people think they *NEED* MI. They can 
>>>always stick with C++.
>>>
>> Or you could find some other language instead of Java? You'd much 
>>rather inconvenience all the other Java programmers by removing 
>>features they need, rather than convenience yourself by finding 
>>something that suits you better?
>
>No, you're messing up the metaphor. I'm proposing that the next version 
>of ANTLR (or maybe it's a completely new tool)
>be to today's ANTLR as Java is to C++: a new and improved version 
>that's really better, in part because it takes programming
>to a higher level.
>
If you want to write a new tool, go ahead. But it won't be Antlr. And 
the people who know Antlr will go on using it because it's better :-)
>>>>>>
>>>
>>>> Let's ask a question ... how can a tool be "great" if it *relies* 
>>>>other tools even for its existence?
>>>
>>>
>>> Sure, of course! Virtually all software relies on other tools 
>>>(compilers, operating systems, etc) for their existence.
>>>
>>>>
>>>> A C compiler can compile itself. Can a Java system build itself? 
>>>>How much of Java is actually written in Java? (Oh and I'm including 
>>>>the supporting libraries here!)
>>>
>>>
>>> Almost all of Java is written in Java. The only part that's not is 
>>>the lowest layer, which is OS and hardware specific.
>>>
>> Assembler is written in assembler, pretty much without exception. C 
>>is written in C, pretty much without exception.
>>
>> The reason for demanding that a great tool be written in itself is 
>>exemplified by my databasic example above - the primitives for the 
>>version I initially used were written in machine code, and followed 
>>the spec. The C rewrite introduced loads of subtle bugs, because the 
>>characteristics of the C environment were different. How much of Java 
>>DEPENDS on the lowest layer? How much knowledge do you need to 
>>understand that layer?
>
>Heheh...didn't know that most of Java was written in Java, did you? ;)
>Yes, all of Java DEPENDS on C.  All of C DEPENDS on object code. All of 
>object code DEPENDS on
>the operating system. All of the operating system DEPENDS on the hardware.
>
Can you separate DEPENDS and IS? Java *depends* on C because at its base 
it's written in C. Neither Java nor C *depend* on object code, because 
at base they *are* object code. (Java twice over, because it's JVM 
object code interpreted by a JVM which is CPU object code :-)

Oh. And I've never met object code that depends on the operating system. 
On the hardware, yes, because it relies on the CPU to execute it 
correctly. But the OS? Not at all.

Actually, I'm beginning to think this is part of your problem. You've 
used the word DEPENDS four times in the paragraph above. With, to my 
mind, at least three different meanings! Everybody's trying to help you, 
but we don't seem able to understand what you're trying to say.
>>>>
>>>> Antlr v3 is due to be rewritten in Antlr v3. To my mind, that's a 
>>>>"necessary but not sufficient" condition for greatness.
>>>
>>>
>>> Wow. Only software that's written in itself can be "great"? Are you 
>>>really saying that?
>>>
>> Yes I am. Because otherwise it requires you to understand much more 
>>than you want about the execution environment. Note my comments about 
>>subtle bugs. Because it requires the system guys to be *experts* in 
>>two completely different environments!
>>
>> I note that you think a "great tool" is one that means you *don't* 
>>*need* to be an expert. By your logic, Java therefore has to be a crap 
>>tool for Java system programmers ...
>
>No, I don't mean "...*dont need* to be an expert", but rather "dont 
>need to know about the layers below". Java is great
>because it doesn't require any knowledge about memory management, for 
>example. The tool should let you
>only be an expert *at one level of abstraction*. I can program in Java 
>without knowing (almost) any details about
>any of the underlying layers.
>
Welcome to your imaginary world! By training, I'm a scientist. Your 
world just doesn't exist in reality! In order to be an expert at, say, 
Chemistry (my speciality) you need to know a hell of a lot about 
Physics. And depending on your branch of Chemistry, you also almost 
certainly need to know a hell of a lot about Biology, or Geology, or 
loads of other stuff.
>>
>> Which is why, in the system I want to build with Antlr, it's very 
>>important to me that AS MUCH AS POSSIBLE of the system is built using 
>>DATABASIC, because that's what the programmers using it will have 
>>experience in. *THAT* is why a great tool must be built using itself.
>
>I think maybe you're only picturing cases where the tool developer is 
>the same person as the tool user.
>I don't care at all whether ANTLR (Or Java, or anything else I use) is 
>built with itself. I'm sure Terence cares,
>but he's the exception.
>
You should care. Otherwise you'll get bitten by paradigm bugs.
>>
>>>
>>> Well, virtually every coder in the world is like that. Only a 
>>>handful  of people in the world know the details of what their
>>> compiler is doing as they use it every day. Same goes for the OS and 
>>>the hardware. Same goes for  the car you drive
>>> and just about everything other non-trivial machine you use: you use 
>>>it without knowing much about how it works.
>>> That's what makes the world go round :)
>>>
>> You miss the point. Let's take the car example. There's a big 
>>difference between what I wrote and you responded to. As far as I can 
>>see, you're saying that if a guy has a driving licence, you're quite 
>>happy to get him to repair your car brakes ... WHAT...!!!
>
>No, I'm just saying that what makes the car useful is that you don't 
>have to understand it to use it. You're
>hidden from the layers below.
>
And if you don't have basic mechanical knowledge you are a liability and 
a danger to other road users. The UK driving test now includes 
elementary car maintenance as a mandatory component. Fail that, and you 
fail your test.

And as I said above, to be an expert you need a considerable amount of 
knowledge in realms close to your specialist subject ... if that's true 
in the scientific world, I doubt it's any different in the programming 
world.
>>
>> You do not give a problem to someone to solve, if they do not have 
>>the mental capacity to understand the problem. You do not ask a guy to 
>>repair the brakes on your car if he has no experience of being a 
>>mechanic, even if he does have a driving licence (and there's a lot of 
>>people I would hate to have service my car, even if they do have a 
>>mechanics qualification, because they learnt to follow the diagrams in 
>>the book and didn't learn to work through what was *actually* *happening*!).
>
>I agree.
>
>>>>>
>>>> And then the programmer needs a pointer to write to a bit of 
>>>>hardware, and throws Java out in disgust because his "great" tool is 
>>>>useless for the job at hand ...
>>>
>>>
>>> Right. And probably gets fired if his boss finds out that he's 
>>>trying  to write a bit directly to hardware.
>>> He thinks he needs to do that, and he thinks he's smart, but he's 
>>>not. That's pretty rare these days to
>>> have anyone feel that they need direct access to the hardware. We're 
>>>many levels beyond that now.
>>>
>> Given the ability of many bosses, I bet the boss gave him the job, 
>>and told him to use the wrong tool ...
>
>Maybe. From my experience, almost always when someone says "I need to 
>set individual bits by hand,
>so Java isn't a good fit", it turns out that in fact he does not need 
>to set individual bits by hand, and Java will
>work just fine. He just thinks he needs to bit-twiddle because that's 
>how he's used to doing things, and he
>wouldn't even think to just use an array of booleans because he can't 
>make that mental leap.
>
Or maybe, just maybe, an array of booleans is a crap fit for the job. I 
don't know how Java stores booleans internally - I know C stores one 
boolean per byte. Crap, because it wastes a hell of a lot of (maybe) 
precious memory. Maybe Java's crap the same way, maybe not.

I've used PL/1. That stores booleans efficiently (as bits). But it bit 
(sorry for the pun) many programmers, because it went from most to least 
significant. Pass "true" between PL/1 and FORTRAN and the result was a 
disaster because FORTRAN, iirc, went from least to most - anyway, on the 
system I used there were big red warnings to be careful everywhere.
>>
>>>>
>>>> Because javac is meant to convert java source accurately to java 
>>>>bytecode. While Antlr is meant to (and does) accurately convert its 
>>>>sourcecode into a lexer/parser/treewalker, it is also designed to 
>>>>let do things beyond the power of the tool. Both fit their design 
>>>>aims  - javac creates a java program, Antlr creates an extensible 
>>>>lexer/parser/treewalker. Both are *good* tools, that doesn't 
>>>>necessarily mean they are *great* tools.
>>>>
>>>> At the end of the day, I think your definition of "great" is badly 
>>>>flawed - "a great tool lets a mediocre programmer do a decent job". 
>>>>A great programmer could probably outperform that mediocre 
>>>>programmer without that tool.
>>>
>>>
>>> Heh, it's been a while since I heard the "A great programmer can 
>>>write better code than the compiler" line.
>>> Used to hear that a lot 20 years ago, not much any more.
>>
>>
>> I gather the statistics still hold true. The best programmers still 
>>outperform the average ones by about 1000% ... that's a MASSIVE 
>>difference.
>
>Yea, that's about right. But the best compilers now outperform the best 
>programmers at compiling. We're
>comparing people to tools above, not people to people.
>
Forth, anyone?

But, to get back to Antlr, Antlr is a compiler for grammars. There are 
things we cannot tell Antlr how to do, because we don't have the 
knowledge to be able to specify it. So Antlr is the best compiler 
possible, but it can't do the full job. So we design Antlr so that 
people can build on what Antlr has done ...
>>
>>>
>>>>
>>>> The greater the tool, the greater the level of maturity required to 
>>>>able to use it safely.
>>>
>>>
>>> If you want to use that meaning for "great", that's fine. But I do 
>>>disagree. If you had one space shuttle
>>> that was completely safe and automated, that would be a "greater" 
>>>tool than one that requires maturity, IMO.
>>> Especially if you have millions of shuttle pilots, and not just a few.
>>
>>
>> You've obviously not read Dick Feynmann then ... :-) even *I* could 
>>fly the space shuttle :-)
>
>"Surely You're Joking!" ;) Maybe you could point me to where he talks 
>about this issue,
>because I don't remember it.
>
Well, the take-off is all done by George. Flying it in space is just a 
matter of pointing it where you want to go. And the landing was mostly 
done by George - the only thing the pilot (used to) did was press the 
red button to lower the landing gear.

And Dick said "Whatever for? Either let the guy fly the thing properly, 
or take responsibility for it away completely".

So, basically, while the pilot is also the captain of the shuttle, his 
job is 99.99% being captain, and 0.01% being pilot ... being pilot means 
sitting back and letting George do the job.
>>
>>>
>> Fine. Since Java is such a great language, let's see you write the 
>>BIOS for all new computers in it ... what? you mean all new computers 
>>will be bricks if you do? What a crap tool ...
>
>Just because a car can't fly doesn't mean it's not a great tool. ;)
>
>>
>> Oh - and as for "we smart programmers", you do realise you've just 
>>disqualified yourself as a member of that set? You used the word "we"! 
>>If the "other" guy respects you, he'll do as you say. I've done some 
>>crazy things in my time, like using a goto in C :-) But when other 
>>programmers have said "you shouldn't do that" I've just said "well, 
>>you do better". Usually, they've looked at the problem to be solved, 
>>and backed down. If you're a smart programmer, you'll show the other 
>>guy how to do it better. If you can't, he's smarter than you thought!
>
>I'll take that challenge...show me a piece of code that really needs a 
>goto, and I'll remove it (assuming you let me rewrite
>the whole darn thing if I need to).
>
I won't let you rewrite the spec, because I didn't write that. It 
contained two fairly convoluted flow charts, with a goto jumping across 
from one path into the other. Basically, an outer if/then/else, and 
then, at just one point half way down the if, under a certain condition 
it jumped across into the else.

I have no doubt you could refactor the code to remove the goto. But you 
would have quadrupled the size of the source, and destroyed any 
resemblance to the specification. And imho, clarity of the source and 
visible adherence to the client spec are much more important than 
"theoretical purity". Oh - and despite the goto, the spec was provably 
correct and the simplest definition of the client's problem.
>>
>> "Perfection is achieved, not when everything that is necessary has 
>>been added, but when everything that is unnecessary has been taken 
>>away".
>
>And so it is with C++ and Java.
>
>>
>>
>> I *know* I'm an expert in DATABASIC. I would *like* to *think* I'm a 
>>guru, but until somebody like Jim, or Rob, or Monty, tells me I am 
>>(and I probably wouldn't believe Monty), then I'm not! And even then, 
>>I wouldn't believe them unless other people I trusted agreed with them.
>>
>> Ask Loring what unnecessary stuff can be removed from Antlr. Ask Ter. 
>>Ask Jim. Ask Monty. Ask the people *YOU* *RESPECT* *AS* *EXPERTS* (or 
>>aren't there any such on this list? :-)
>
>Ask Bjourne what unnecessary stuff can be removed from C++. Now ask 
>Gosling. Often, the experts are the
>exact wrong people to ask.
>
I went on about "completeness". Yes I know Godel's theorem (well, sort 
of), but a lot of things may be like the top quark, or the Higgs Boson. 
As far as we know, they don't exist and we don't need them, but we need 
a place for them to exist or the theory is incomplete. The reason C++ is 
bigger than Java may be because the extra stuff is "needed" for 
completeness. I strongly suspect it is.

Programmers are very poor at seeing the "problem space". That's why so 
many programs crash when they hit an error. Those features you see as 
"superfluous" may well exist because Bjarne (spell his name right - it 
took me several attempts before I realised who you were talking about) 
mapped out the problem space, and there was a feature-shaped hole.

I've regularly come across "neat features" that someone wants to add to 
a tool, and my immediate reaction is "you don't understand the 
philosophy behind the tool. That feature doesn't exist because it 
doesn't belong!". I'm not saying those sort of features haven't made 
their way into C++ or Java. But it wouldn't surprise me if many features 
are there because they DO belong, even if they LOOK superfluous.

And Antlr looks to me as though Ter *has* thought things through, and it 
has the features it has precisely because they DO belong, whether they 
seem superfluous or not. And if they're not what you want, it's because 
you've picked the wrong tool for the job.
>>
>> I think you'll find everything in Antlr is there because it's needed. 
>>Like ASTs for example - I think they happen to be a perfect solution 
>>to my problem :-)
>>
>> I know Antlr doesn't do what you want. So stop trying to drive screws 
>>with a hammer, and go find yourself a screwdriver :-) Don't try and 
>>ban hammers :-)
>
>You do understand that I'm not trying to "disable" or "take away 
>features" from ANTLR, right?
>I think it's perfectly reasonable for someone to post his crazy dreams 
>for an "electric hammer" on
>a "Joe's hammer" forum. If you don't want to read it, then don't.
>
Then why go on about a possible future Antlr v4, that isn't "Antlr as we 
know it"?

Cheers,
Wol
-- 
Anthony W. Youngman - anthony at thewolery.demon.co.uk