[antlr-interest] lexer woes

[Loring Craymer]

Say rather that fragments should be called with care for the time being than that they are of dubious utility.  There is no problem in calling fragments as tail calls or when there is no ambiguity.  This is basically a "growing pains" problem--solutions which fix this one are likely to enable better lexer analysis and make it easy to provide lexer states (at least the mTokens part of the problem).

--Loring

----- Original Message ----
> From: Matt Benson <gudnabrsam at yahoo.com>
> To: Loring Craymer <lgcraymer at yahoo.com>; Gavin Lambert <antlr at mirality.co.nz>; Antlr List <antlr-interest at antlr.org>
> Sent: Thursday, March 6, 2008 8:03:30 AM
> Subject: Re: [antlr-interest] lexer woes
> 
> 
> --- Loring Craymer  wrote:
> 
> > There is also some significant analysis to remove
> > BACKTRACK_SYNPREDs when they are not needed.  For
> > this case, they would not be needed (LL* is
> > sufficient), so they are removed and you are back to
> > the FOLLOW set problem.
> 
> Okay... I definitely need to keep reading Dragon if I
> want to actually understand the FOLLOW set problem;
> however, going back to your first response, that's
> just how it goes, fragments are of dubious utility for
> the time being and as long as I bear that in mind I
> just might be able to come up with a properly
> functioning, if less-than-aesthetically-specified,
> lexer?  Does that sum it up?  Thanks for all your
> attention here, L.
> 
> -Matt
> 
> > 
> > --Loring
> > 
> > 
> > 
> > ----- Original Message ----
> > > From: Matt Benson 
> > > To: Loring Craymer ; Gavin
> > Lambert ; Antlr List
> > 
> > > Sent: Wednesday, March 5, 2008 3:55:13 PM
> > > Subject: Re: [antlr-interest] lexer woes
> > > 
> > > 
> > > --- Loring Craymer  wrote:
> > > 
> > > > Setting backtracking in combined grammars only
> > > > affects the parser (again, see the
> > prefixWithSynPred
> > > > code); from the code, though, setting
> > > > backtracking=true in lexer grammars does
> > introduce
> > > > synpreds at all decision points.
> > > > 
> > > 
> > > From the code... but try to tool this:
> > > ----
> > > lexer grammar EasyLexer;
> > > options {
> > >     backtrack=true;
> > >     memoize=true;
> > > }
> > > 
> > > Ident
> > >     :    Letter (Letter|Digit)*
> > >     ;
> > > 
> > > fragment
> > > Letter 
> > >     :    'A'..'Z'|'a'..'z'|'_'
> > >     ;
> > > 
> > > fragment
> > > Digit
> > >     :    '0'..'9'
> > >     ;
> > > 
> > > ---
> > > 
> > > Does it work, or have I had too much crack?
> > > 
> > > -Matt
> > > 
> > > > --Loring
> > > > 
> > > > ----- Original Message ----
> > > > > From: Gavin Lambert 
> > > > > To: Loring Craymer ; Matt
> > > > Benson ; Antlr List
> > > > 
> > > > > Sent: Wednesday, March 5, 2008 12:09:41 AM
> > > > > Subject: Re: [antlr-interest] lexer woes
> > > > > 
> > > > > At 11:35 5/03/2008, Loring Craymer wrote:
> > > > >  >> 1) Whether backtracking mode is supported
> > for
> > > > lexers,
> > > > >  >1.)  Yes--see calls to prefixWithSynPred()
> > in
> > > > antlr.g
> > > > > 
> > > > > I think that answer is a little misleading :)
> > > > > 
> > > > > As far as I know, the "backtrack=true" option
> > has
> > > > no effect in the 
> > > > > lexer.  However, backtracking in general is
> > > > supported through 
> > > > > adding your own syntactic predicates.
> > > > > 
> > > > > Note that syntactic predicates can only be
> > used to
> > > > disambiguate 
> > > > > between alts in a single rule -- you can't use
> > > > them to distinguish 
> > > > > between tokens at the top level.  (If you need
> > to
> > > > do that, you'll 
> > > > > have to merge the rules.)
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > >      
> > > >
> > >
> >
> ____________________________________________________________________________________
> > > > Never miss a thing.  Make Yahoo your home page. 
> > > > http://www.yahoo.com/r/hs
> > > > 
> > > 
> > > 
> > > 
> > >       
> > >
> >
> ____________________________________________________________________________________
> > > Never miss a thing.  Make Yahoo your home page. 
> > > http://www.yahoo.com/r/hs
> > > 
> > 
> > 
> > 
> > 
> >      
> >
> ____________________________________________________________________________________
> > Looking for last minute shopping deals?  
> > Find them fast with Yahoo! Search. 
> >
> http://tools.search.yahoo.com/newsearch/category.php?category=shopping
> > 
> 
> 
> 
>       
> ____________________________________________________________________________________
> Be a better friend, newshound, and 
> know-it-all with Yahoo! Mobile.  Try it now.  
> http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ 
> 
> 




      ____________________________________________________________________________________
Looking for last minute shopping deals?  
Find them fast with Yahoo! Search.  http://tools.search.yahoo.com/newsearch/category.php?category=shopping