[antlr-interest] Non-determinism (was: Can I force a token to have precendence in the lexer?)

[Andy Hull]

Wow, thanks for the article. I was able to redefine the language to avoid the problem in order to keep the parser as simple as possible (now using "to" instead of "..." ).

My parser needs to be able to handle nested array expressions like so

{1,2,{5 to 10}, {3,6,9}, 4}

I have the following grammar:

arrayExpression
:   LEFT_BRACKET! arrayInitializer? RIGHT_BRACKET!;
arrayInitializer
:  (e+=expression (',' e+=expression)*)+ -> ^(ELEMENTLIST $e*)
|  expression AUTO expression -> ^(AUTO expression expression)
;

expression
: arrayExpression
/* | other types of expression */
;

with the expected non-LL(*) grammar because "arrayInitializer" depends on the recursive rule expression. Setting backtrack to true doesn't resolve this as I expected.

x={1,2,3,4};

yields the correct tree but...

x={1 to 3};

yields the error:

BR.recoverFromMismatchedToken
line 1:5 mismatched input 'to' expecting RIGHT_BRACKET

arrayInitializer behaves as expected when it contains only a single subrule (either the element list or the range initializer).

Is backtracking the right solution to the non-determinism? I am doing something wrong?

Thanks!

On Apr 20, 2010, at 2:48 PM, Jim Idle wrote:

This examples does everything you need and more, so you can pare it down to just your examples.

http://www.antlr.org/wiki/display/ANTLR3/Lexer+grammar+for+floating+point%2C+dot%2C+range%2C+time+specs


Jim

-----Original Message-----
From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
bounces at antlr.org] On Behalf Of Andy Hull
Sent: Tuesday, April 20, 2010 2:42 PM
To: antlr-interest at antlr.org
Subject: [antlr-interest] Can I force a token to have precendence in
the lexer?

I have the following lexer tokens defined:

I have the following expression "x={1..3};"

I'm expecting the input between the braces to be recognized as FLOAT,
AUTO, FLOAT but what I actually get is just a FLOAT  ".3" in the token
stream of "x","=","{",".3","}",";"


AUTO  :  '...';

FLOAT
   :   ('0' .. '9')+ '.' ('0' .. '9')+ EXPONENT?
   |   '.' ( '0' .. '9' )+ EXPONENT?
   |   ('0' .. '9')+ EXPONENT
   |   ('0' .. '9')+
   ;

I think the float is matching too much. Can anyone help?

Thanks
--
Andy Hull | Senior Software Engineer
SunRun Inc.<http://www.sunrunhome.com/>
direct  415.684.9851

Review SunRun on Yelp<http://www.yelp.com/biz/sunrun-san-francisco>
click on the link above and then on 'write a review'


List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
email-address




List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address

--
Andy Hull | Senior Software Engineer
SunRun Inc.<http://www.sunrunhome.com/>
direct  415.684.9851

clean solar energy without the high upfront costs<http://www.sunrunhome.com/>