[antlr-interest] Re: Why is this nondeterministic?

[mzukowski at yci.com]

Antlr lexers never know what is coming next and don't preserve state between
calls to nextToken().  You can get around this somewhat with multiple
lexers, but I don't see any advantage using that for this problem.  

In your case you could get by with putting the rules together and changing
the token type.  That way antlr looking ahead to the next token won't get
confused.

REGULAR: (REGULAR_BIT)+;
protected SPECIAL: 'a' 'b';

protected
REGULAR_BIT:
    ('b' | 'c')
|   ('a' 'a')
|   ('a' 'c')
|   SPECIAL {$setType(SPECIAL);}
;

Monty

> -----Original Message-----
> From: danfuzz [mailto:danfuzz at milk.com]
> Sent: Monday, May 27, 2002 3:00 PM
> To: antlr-interest at yahoogroups.com
> Subject: [antlr-interest] Re: Why is this nondeterministic?
> 
> 
> Terence writes:
> >The problem is that 'a' 'b' can start both REGULAR (via REGULAR_BIT) 
> >and SPECIAL.  The lexer, given "ab" input would not know which token 
> >to make.  This is an ambiguous grammar, which implies that it is 
> >also nondeterministic.
> 
> Thanks for the quick reply. I'm still confused, though. I thought I
> defined things such that REGULAR *can't* start with 'a' 'b'
> (given that REGULAR_BIT matches 'a' 'a' and 'a' 'c' but not
> 'a' 'b').
> 
> My intent is for the lexer to return alternating tokens of
> REGULAR or SPECIAL, where the text of REGULAR is a run of 'b's and 
> 'c's interspersed with "aa" and "ac" pairs (that is, 
> "(b|c|aa|ac)*" in regex terms), and where the text of SPECIAL is 
> always "ab".
> 
> Is it possible to say this with ANTLR? I noticed that, without
> REGULAR being a closure (that is, saying just "REGULAR: 
> REGULAR_BIT;") that the grammar compiles without complaint, so my
> current workaround is to accumulate the REGULAR string  outside the 
> lexer.
> 
> -dan
> 
> #####
> 
> Recap:
> 
> REGULAR: (REGULAR_BIT)+;
> SPECIAL: 'a' 'b';
> 
> protected
> REGULAR_BIT:
>     ('b' | 'c')
> |   ('a' 'a')
> |   ('a' 'c')
> ;
> 
> 
>  
> 
> Your use of Yahoo! Groups is subject to 
> http://docs.yahoo.com/info/terms/ 
> 
> 
> 

 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/