[antlr-interest] Re: lexical nondeterminism warning

[Xue Yong Zhi]

Your snippet is fine. I even tried it:

1. Create test.g:
class L extends Lexer;
options
{
	exportVocab=Test;
	k = 3;
	charVocabulary='\u0003'..'\uFFFF';
}

LT: '<';     // the less than character
LE : "<=" | "=<" ;   // less than or equal to
NEQ : "<>" ;   // Not equal to
END : "</script>";  // end of script tag
COMMENT: "<!-" ; // HTML comment begin


2. Run it:
$ java -classpath antlr.jar antlr.Tool test.g
ANTLR Parser Generator   Version 2.7.6 (2005-12-22)   1989-2005


No warning at all. Would you check the grammar and see if there is other 
rules/options?


tinker wrote:
> Hi,
>    I am trying to define the lexer for a scripting language that has
> the following constructs:
>    LT: '<';     // the less than character
>    LE : "<=" | "=<" ;   // less than or equal to
>    NEQ : "<>" ;   // Not equal to
>    END : "</script>";  // end of script tag
>    COMMENT: "<!-" ; // HTML comment begin
> 
>   When I tried to process the above grammar file with antlr (v2.7.6)
> with lookahead(k) set to 3, I got 4 lexical nondeterminism warnings,
> between COMMENT and each of LT, LE, NEQ and END.
>     I rewrote LT as following to eliminate the lexical nondeterminism
> between it and  COMMENT -
>    LT: {!(LA(2)=='!' && LA(3)=='-')}?'<' ; //match LT only when it's
> not followed by "!-"
> 
>   However, i am still left with the following three warnings:
> =========================================
> warning:lexical nondeterminism between rules LE and COMMENT upon
>      k==1:'<'
>      k==2:'<','='
>      k==3:<end-of-token>
> warning:lexical nondeterminism between rules NEQ and COMMENT upon
>      k==1:'<'
>      k==2:'>'
>      k==3:<end-of-token>
> warning:lexical nondeterminism between rules END and COMMENT upon
>      k==1:'<'
>      k==2:'/'
>      k==3:'s'
> =========================================
> 
> I was wondering what was the recommended way of dealing with such
> warnings? Is there any way to rewrite the definitions so that the
> warnings go away? Should I add the conditional statements (if
> LA(2)....) to each rule?
> 
> Any help you can provide would be greatly appreciated.
> Thanks,
> T
>