[antlr-interest] MismatchedTokenException and how to find errors in ANTLRWorks

Guntis Ozols guntiso at latnet.lv
Tue Feb 12 15:58:27 PST 2008


>  >Seriously, Jim:  what's the likelihood of your (or
>  >anyone's) being able to sum up the reason for this
>  >fairly succinctly?  What is the #1 gotcha about using
>  >literals vs. tokens that ANTLR3 beginners overlook?
>
> I gave my answer to that question yesterday (though more
> briefly).  As I see it, there are two big drawbacks to using
> literal strings in the parser:
>
> 1. The generated code becomes filled with references to "T32" and
> "T63" etc rather than anything meaningful, making it harder to
> understand.  (And also complicating things if you want to generate
> an AST later on.)
>
> 2. It's too easy to forget that each quoted string produces a
> lexer rule (which may conflict with other rules), making it harder
> to find the source of ambiguity problems.  In addition, "'x' 'y'"
> is not the same as "'xy'" in the parser, though it is in the
> lexer; this can cause confusion.

Sure it is confusing. I fell into this 'xy' trap, too.
If COLON is defined and STAR is defined, then creating
another token for ':*' in a *parser* rule is confusing.
It should be fixed in antlr.
At least, antlr should emit 'anonymous token' warning[s].

I prefer literals because of simplicity / readability.
The tool needs to be patched, not users.

Guntis



More information about the antlr-interest mailing list