[antlr-interest] Re: Advantage of where Token is defined?
lgcraymer
lgc at mail1.jpl.nasa.gov
Mon May 5 10:59:41 PDT 2003
Defining literals as tokens should not affect lexical
nondeterminism. The cure is to do at least one of the following:
1.) Increase k
2.) Refactor--you have at least two definitions with overlapping
character sequences. (Basic cause of lexical nondeterminism).
3.) Use predicates to disambiguate alternatives.
--Loring
--- In antlr-interest at yahoogroups.com, "cgodfrey86" <cgodfrey at e...>
wrote:
> Are there any advantages/disadvantages to defining a token in the
> tokens section of the lexer definition verses as a separate item? I
> found myself doing this to try and avoid "lexical nondeterminism"
> messages, but I am not sure of the implications in doing this.
>
> Thanks.
>
> options
> {
> language = "CSharp";
> }
>
> class ASPULexer extends Lexer;
> options {
> k=3;
> caseSensitive=false;
> }
>
> tokens {
> AND = "and";
> ....
> }
>
>
> instead of
>
>
> AND : "and";
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
More information about the antlr-interest
mailing list