[antlr-interest] Re: lexical nondeterminism

[Mark]

INTEGER_LITERAL: DECIMAL_NUMBER_LITERAL | HEX_NUMBER_LITERAL |
OCTAL_NUMBER_LITERAL;

FLOATING_POINT_LITERAL:
(DIGIT)+ '.' (DIGIT)* (EXPONENT_PART)? (FLOAT_TYPE_SUFFIX)? |
'.' (DIGIT)+ (EXPONENT_PART)? (FLOAT_TYPE_SUFFIX)? |
(DIGIT)+ EXPONENT_PART (FLOAT_TYPE_SUFFIX)? |
(DIGIT)+ (EXPONENT_PART)? FLOAT_TYPE_SUFFIX;

> I saw, a similar question was discussed already yesterday. I searched for the wrong 
keywords in the archive so I did not find these messages. Sorry for that.
> 
> On the other hand however I tried the supposed solution and found it not working (it 
was as if I had not specified the greedy option). I also read the help and tried to 
understand what I can do, but still have no idea.

Yes - that was my question... But your case and my case are different because in your 
case, the common prefix of both INTEGER_LITERAL and FLOATING_POINT_LITERAL is 
(DIGIT)+.  The common prefix is arbitrarily long.  There is no fixed k value long enough.  
The parser would have to look ahead an arbitrary number of characters (until it hit some 
non-digit) before it could decide to go down the INTEGER_LITERAL path or the 
FLOATING_POINT_LITERAL path.

In my case, the common prefix was only one character and k was set to 2, so the lexer 
could indeed predict with fixed look-ahead.  My 'error' turned out to just be an overly 
conservative warning from the lexer, and the greedy option simply silenced the message.  
The generated code was always correct and deterministic.

Look in the examples folder of the Antlr distribution at the pascal or java grammars to 
see how to handle lexar matching of numbers.  Basically, you have to match the prefix 
(DIGIT)+ in one rule, and then based on what matches next, use {$setType(...)} to fix-up 
the token type.

- Mark




 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/