[antlr-interest] lexer woes

Matt Benson gudnabrsam at yahoo.com
Thu Mar 6 08:03:30 PST 2008


--- Loring Craymer <lgcraymer at yahoo.com> wrote:

> There is also some significant analysis to remove
> BACKTRACK_SYNPREDs when they are not needed.  For
> this case, they would not be needed (LL* is
> sufficient), so they are removed and you are back to
> the FOLLOW set problem.

Okay... I definitely need to keep reading Dragon if I
want to actually understand the FOLLOW set problem;
however, going back to your first response, that's
just how it goes, fragments are of dubious utility for
the time being and as long as I bear that in mind I
just might be able to come up with a properly
functioning, if less-than-aesthetically-specified,
lexer?  Does that sum it up?  Thanks for all your
attention here, L.

-Matt

> 
> --Loring
> 
> 
> 
> ----- Original Message ----
> > From: Matt Benson <gudnabrsam at yahoo.com>
> > To: Loring Craymer <lgcraymer at yahoo.com>; Gavin
> Lambert <antlr at mirality.co.nz>; Antlr List
> <antlr-interest at antlr.org>
> > Sent: Wednesday, March 5, 2008 3:55:13 PM
> > Subject: Re: [antlr-interest] lexer woes
> > 
> > 
> > --- Loring Craymer  wrote:
> > 
> > > Setting backtracking in combined grammars only
> > > affects the parser (again, see the
> prefixWithSynPred
> > > code); from the code, though, setting
> > > backtracking=true in lexer grammars does
> introduce
> > > synpreds at all decision points.
> > > 
> > 
> > From the code... but try to tool this:
> > ----
> > lexer grammar EasyLexer;
> > options {
> >     backtrack=true;
> >     memoize=true;
> > }
> > 
> > Ident
> >     :    Letter (Letter|Digit)*
> >     ;
> > 
> > fragment
> > Letter 
> >     :    'A'..'Z'|'a'..'z'|'_'
> >     ;
> > 
> > fragment
> > Digit
> >     :    '0'..'9'
> >     ;
> > 
> > ---
> > 
> > Does it work, or have I had too much crack?
> > 
> > -Matt
> > 
> > > --Loring
> > > 
> > > ----- Original Message ----
> > > > From: Gavin Lambert 
> > > > To: Loring Craymer ; Matt
> > > Benson ; Antlr List
> > > 
> > > > Sent: Wednesday, March 5, 2008 12:09:41 AM
> > > > Subject: Re: [antlr-interest] lexer woes
> > > > 
> > > > At 11:35 5/03/2008, Loring Craymer wrote:
> > > >  >> 1) Whether backtracking mode is supported
> for
> > > lexers,
> > > >  >1.)  Yes--see calls to prefixWithSynPred()
> in
> > > antlr.g
> > > > 
> > > > I think that answer is a little misleading :)
> > > > 
> > > > As far as I know, the "backtrack=true" option
> has
> > > no effect in the 
> > > > lexer.  However, backtracking in general is
> > > supported through 
> > > > adding your own syntactic predicates.
> > > > 
> > > > Note that syntactic predicates can only be
> used to
> > > disambiguate 
> > > > between alts in a single rule -- you can't use
> > > them to distinguish 
> > > > between tokens at the top level.  (If you need
> to
> > > do that, you'll 
> > > > have to merge the rules.)
> > > > 
> > > > 
> > > 
> > > 
> > > 
> > > 
> > >      
> > >
> >
>
____________________________________________________________________________________
> > > Never miss a thing.  Make Yahoo your home page. 
> > > http://www.yahoo.com/r/hs
> > > 
> > 
> > 
> > 
> >       
> >
>
____________________________________________________________________________________
> > Never miss a thing.  Make Yahoo your home page. 
> > http://www.yahoo.com/r/hs
> > 
> 
> 
> 
> 
>      
>
____________________________________________________________________________________
> Looking for last minute shopping deals?  
> Find them fast with Yahoo! Search. 
>
http://tools.search.yahoo.com/newsearch/category.php?category=shopping
> 



      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ 



More information about the antlr-interest mailing list