[antlr-interest] intermittent problem with lexer

[Thomas Brandon]

On Mon, Mar 10, 2008 at 12:23 PM, Sean Proctor <sproctor at gmail.com> wrote:
> Hello,
>
> I've been using ANTLR in one of my projects for about 6 months. This problem
> first started with some changes that I made 2 or 3 weeks ago. Removing the
> old files and rerunning the build tool seemed to resolve it, but that
> doesn't always.
>
> Here is the error when I get it
>
>     [java] ANTLR Parser Generator  Version 3.0.1 (August 13, 2007)
> 1989-2007
>      [java] warning(205): cc/warlock/core/stormfront/script/wsl/WSL.g3:1:8:
> ANTLR could not analyze this decision in rule Tokens; often this is because
> of recursive rule references visible from the left edge of alternatives.
> ANTLR will re-analyze the decision with a fixed lookahead of k=1.  Consider
> using "options {k=1;}" for that decision and possibly adding a syntactic
> predicate.
>       [java] warning(209):
> cc/warlock/core/stormfront/script/wsl/WSL.g3:15:1: Multiple token rules can
> match input such as "'<'": T33, T35, T37, STRING
>      [java] As a result, tokens(s) STRING,T37,T35 were disabled for that
> input
>       [java] warning(209):
> cc/warlock/core/stormfront/script/wsl/WSL.g3:16:1: Multiple token rules can
> match input such as "'>'": T34, T36, STRING
>      [java] As a result, tokens(s) STRING,T36 were disabled for that input
>       [java] warning(208):
> cc/warlock/core/stormfront/script/wsl/WSL.g3:17:1: The following token
> definitions are unreachable: T35,T36,T37
>
> I tried using an intermediate build from 2-27. The first warning went away
> and the last one turned into an error. With the newer build, it seems that
> the error never goes away.
Sound like the first warning may have been due to a bug\limitation in
the older analysis code that has been fixed in the intermediate build.
The intermediate build has also elevated unreachable tokens to being
an error. You need to fix that. '<' and '>' (from the parser literals)
are clashing with your STRING rule. Which should it be? You do realise
that parser context does not affect the lexer and that parser literals
simply generate lexer rules right?
I think you might also want to reconsider your heavy use of states if
possible. Thats going to really complicate your lexer DFAs which may
be what's causing the first warning. You also might end up running
into code size problems and performance will be worse. Is it really
lexically ambiguous in a way you can't deal with with by generalising
the lexer tokenisation and using some setType calls in the parser? A
lot of the stuff seems to deal with things that are only keywords in
certain contexts, there are other ways to do this. See the keywords as
identifiers entry in the wiki.

Tom.
>
> The file in question is located here
> http://fisheye.warlock.cc/browse/Warlock/warlock2/trunk/stormfront/cc.warlock.core.stormfront.script/src/main/cc/warlock/core/stormfront/script/wsl/WSL.g3?r=1693
>
> The language itself is poorly structured, but I didn't create it.
>
> Any suggestions would be greatly appreciated.
>
> Sean
>