[antlr-interest] Re: Advantage of where Token is defined?

[lgcraymer]

Defining literals as tokens should not affect lexical 
nondeterminism.  The cure is to do at least one of the following:
   1.)  Increase k
   2.)  Refactor--you have at least two definitions with overlapping 
character sequences.  (Basic cause of lexical nondeterminism).
   3.)  Use predicates to disambiguate alternatives.

--Loring

--- In antlr-interest at yahoogroups.com, "cgodfrey86" <cgodfrey at e...> 
wrote:
> Are there any advantages/disadvantages to defining a token in the 
> tokens section of the lexer definition verses as a separate item? I 
> found myself doing this to try and avoid "lexical nondeterminism" 
> messages, but I am not sure of the implications in doing this.
> 
> Thanks.
> 
> options
> {
> 	language = "CSharp";
> }
> 
> class ASPULexer extends Lexer;
> options {
>   k=3;
>   caseSensitive=false;
> }
> 
> tokens {
> AND = "and";
> ....
> }
> 
> 
> instead of 
> 
> 
> AND : "and";


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/