[antlr-interest] Fun with ANTLR3: mystery of the huge lexer

[Jim Idle]

> -----Original Message-----
> From: David Piepgrass [mailto:qwertie256 at gmail.com]
> Sent: Saturday, June 30, 2007 5:30 PM
> To: Jim Idle
> Cc: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] Fun with ANTLR3: mystery of the huge
> lexer
> 
> > The lexer will assign the priority from top to bottom in listed
order
> so
> > you should be able to achieve what you need - I find you get simpler
> > lexers though if you code the rule yourself to eliminate clashes and
> use
> > $type = xxx;.
> 
> It turns out they are already in the right order, but the problem is
> these truly gigantic state transition tables. 

I don't think you should get those unless there is something else
slightly wrong with the lexer. Until Ter gets the ANLTR3 recognizer for
ANTLR3 going there are quite a lot of things that probably ought to be
caught that are not. If you don't wish to publish the lexer then if you
want to send it to me off list I would take a look at it out of
interest.

> I may have to combine
> all three rules into one to resolve the problems in a more intimate
> way, but one of the reasons I decided to "tough it out" with ANTLR 3
> (despite a lack of good tutorials*) was I hoped it would be possible
> to avoid such things.

I usually do this because it makes the code come out as simple as
possible (which is always good), but it is more habit than anything - in
general ANTLR is pretty good at getting together a reasonable code size.
I suspect then that there is another rule interfering here. Which
language are you trying to deal with?

> 
> (*Even the ANTLR book isn't as good as this ANTLR 2 tutorial, IMO:)
> http://javadude.com/articles/antlrtut/

I guess it depends on what you are looking for and all that ;-). 

> > User operators eh, how do they specify precedence?
> 
> With numbers. In fact, they can even have a different precedence on
> the left and right sides of a binary operator, as well as making
> ternary and other complex operators. I'm still developing the
> algorithm but I think it will work.

Hmmm - is this language the final language or are you writing a meta
compiler of some sort?

Jim