[antlr-interest] Token class in lexer - lexical nondeterminism error

[Scott Amort]

Hi,

Jiho Han wrote:
> I am new to ANTLR.
> I have the following defined in my lexer.
> 
> OP_EQ		: '=' ;
> OP_NEQ	: "<>" ;
> OP_GT		: '>' ;
> OP_LT		: '<' ;
> OP_GE		: ">=" ;
> OP_LE		: "<=" ;
> LPAREN	: '(' ;
> RPAREN	: ')' ;

You need to preface these rules with 'protected', for example:

protected OP_EQ: '=';

> OPERATOR	: (OP_EQ | OP_NEQ | OP_GT | OP_LT | OP_GE | OP_LE) ;
> 
> When I run it through antlr I get:
> 
> FilterExpression.g: warning:lexical nondeterminism between rules OP_EQ
> and OPERATOR upon
> FilterExpression.g:     k==1:'='
> FilterExpression.g:     k==2:<end-of-token>
> 
> And bunch of others like it.

This is due to non-determinism between any of these expressions (i.e.
are you matching OP_EQ or OPERATOR?)  Using the 'protected' keyword
tells ANTLR that your OP_* rules are helpers, and are not to be passed
on to the parser.

Best,
Scott