[antlr-interest] lexer woes

[Loring Craymer]

There is also some significant analysis to remove BACKTRACK_SYNPREDs when they are not needed.  For this case, they would not be needed (LL* is sufficient), so they are removed and you are back to the FOLLOW set problem.

--Loring



----- Original Message ----
> From: Matt Benson <gudnabrsam at yahoo.com>
> To: Loring Craymer <lgcraymer at yahoo.com>; Gavin Lambert <antlr at mirality.co.nz>; Antlr List <antlr-interest at antlr.org>
> Sent: Wednesday, March 5, 2008 3:55:13 PM
> Subject: Re: [antlr-interest] lexer woes
> 
> 
> --- Loring Craymer  wrote:
> 
> > Setting backtracking in combined grammars only
> > affects the parser (again, see the prefixWithSynPred
> > code); from the code, though, setting
> > backtracking=true in lexer grammars does introduce
> > synpreds at all decision points.
> > 
> 
> From the code... but try to tool this:
> ----
> lexer grammar EasyLexer;
> options {
>     backtrack=true;
>     memoize=true;
> }
> 
> Ident
>     :    Letter (Letter|Digit)*
>     ;
> 
> fragment
> Letter 
>     :    'A'..'Z'|'a'..'z'|'_'
>     ;
> 
> fragment
> Digit
>     :    '0'..'9'
>     ;
> 
> ---
> 
> Does it work, or have I had too much crack?
> 
> -Matt
> 
> > --Loring
> > 
> > ----- Original Message ----
> > > From: Gavin Lambert 
> > > To: Loring Craymer ; Matt
> > Benson ; Antlr List
> > 
> > > Sent: Wednesday, March 5, 2008 12:09:41 AM
> > > Subject: Re: [antlr-interest] lexer woes
> > > 
> > > At 11:35 5/03/2008, Loring Craymer wrote:
> > >  >> 1) Whether backtracking mode is supported for
> > lexers,
> > >  >1.)  Yes--see calls to prefixWithSynPred() in
> > antlr.g
> > > 
> > > I think that answer is a little misleading :)
> > > 
> > > As far as I know, the "backtrack=true" option has
> > no effect in the 
> > > lexer.  However, backtracking in general is
> > supported through 
> > > adding your own syntactic predicates.
> > > 
> > > Note that syntactic predicates can only be used to
> > disambiguate 
> > > between alts in a single rule -- you can't use
> > them to distinguish 
> > > between tokens at the top level.  (If you need to
> > do that, you'll 
> > > have to merge the rules.)
> > > 
> > > 
> > 
> > 
> > 
> > 
> >      
> >
> ____________________________________________________________________________________
> > Never miss a thing.  Make Yahoo your home page. 
> > http://www.yahoo.com/r/hs
> > 
> 
> 
> 
>       
> ____________________________________________________________________________________
> Never miss a thing.  Make Yahoo your home page. 
> http://www.yahoo.com/r/hs
> 




      ____________________________________________________________________________________
Looking for last minute shopping deals?  
Find them fast with Yahoo! Search.  http://tools.search.yahoo.com/newsearch/category.php?category=shopping