[antlr-interest] Token class in lexer - lexical nondeterminism error

james_cataldo at agilent.com james_cataldo at agilent.com
Tue Jul 25 08:44:56 PDT 2006


>  expr: (a:ID op:operator value:LITERAL) ;

What is it that you can't do with this set of labels?  It's okay to add labels to a subrule.


-----Original Message-----
From: james_cataldo at agilent.com [mailto:james_cataldo at agilent.com]
Sent: Tuesday, July 25, 2006 11:22 AM
To: Jiho Han; antlr-interest at antlr.org
Subject: RE: [antlr-interest] Token class in lexer - lexical nondeterminism error

The problem is that if the lexer sees an equals sign, it has two tokens it can create.  One is an OP_EQ token, and the other is an OPERATOR token.  I suggest you make a rule called operator in your parser:

> operator:
>    (OP_EQ | OP_NEQ | OP_GT | OP_LT | OP_GE | OP_LE) ;

Then you shouldn't get any problem about nondeterminism, because tokens will be sent to the parser from the lexer, not characters. Character streams are sent to the lexer, which then creates token streams to send to the parser.  Hope this helps.

Cheers,
Adam

________________________________________
From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Jiho Han
Sent: Tuesday, July 25, 2006 6:16 AM
To: antlr-interest at antlr.org
Subject: [antlr-interest] Token class in lexer - lexical nondeterminism error

I am new to ANTLR. 
I have the following defined in my lexer. 
OP_EQ           : '=' ;
OP_NEQ  : "<>" ;
OP_GT           : '>' ;
OP_LT           : '<' ;
OP_GE           : ">=" ;
OP_LE           : "<=" ;
LPAREN  : '(' ;
RPAREN  : ')' ;
OPERATOR        : (OP_EQ | OP_NEQ | OP_GT | OP_LT | OP_GE | OP_LE) ; When I run it through antlr I get: 
FilterExpression.g: warning:lexical nondeterminism between rules OP_EQ and OPERATOR upon
FilterExpression.g:     k==1:'=' 
FilterExpression.g:     k==2:<end-of-token> And bunch of others like it. 
I tried to create OPERATOR as a token class as mentioned in the antlr documentation in the section titled, Meta Language.
So that I can do this in the parser: 
expr: ID OPERATOR^ VALUELITERAL ;
Instead of,
expr: ID (OP_EQ | OP_NEQ | OP_GT | OP_LT | OP_GE | OP_LE) VALUELITERAL ; What am I missing? 
Thanks
Jiho Han
Senior Software Engineer
Infinity Info Systems
The Sales Technology Experts
Tel: 212.563.4400 x216
Fax: 212.760.0540
jhan at infinityinfo.com
www.infinityinfo.com 



More information about the antlr-interest mailing list