[antlr-interest] MismatchedTokenException and how to find errors in ANTLRWorks
Jim Idle
jimi at temporal-wave.com
Wed Feb 13 09:38:24 PST 2008
> -----Original Message-----
> From: Guntis Ozols [mailto:guntiso at latnet.lv]
> > 2. It's too easy to forget that each quoted string produces a
> > lexer rule (which may conflict with other rules), making it harder
> > to find the source of ambiguity problems. In addition, "'x' 'y'"
> > is not the same as "'xy'" in the parser, though it is in the
> > lexer; this can cause confusion.
>
> Sure it is confusing. I fell into this 'xy' trap, too.
> If COLON is defined and STAR is defined, then creating
> another token for ':*' in a *parser* rule is confusing.
> It should be fixed in antlr.
What should be fixed? 'x' 'y' IS something different to the parser than
the lexer! ':' '*' and ':*' ARE different tokens.
> At least, antlr should emit 'anonymous token' warning[s].
>
This post is a classic example of the problem. I am not trying to single
out the poster, just point out the fact that the literals in the parser
create great confusion when your knowledge of the tool is low, and I
don't think you should use this until you have a better command of the
tool. One certainly should not ask for things to be 'fixed'
There isn't anything wrong with the tool, it is doing what you tell it
to do. The token isnt anonymous, it is called T66 or whatever, the
runtime doesn't know that you created it by specifying it in the parser.
So, I don't see why "anonymous token" would help you at all.
> I prefer literals because of simplicity / readability.
However, you are showing in fact that if anything, it is more
complicated, not simpler. Readability is subjective. Is ':' anymore
readable than COLON once you have the simple rule that words starting
with Upper Case are tokens from the lexer?
> The tool needs to be patched, not users.
A tool should strive to be as simple as possible, but no simpler :-)
While that statement makes sense, you are an under an obligation to
yourself to try to make sure you understand the tool too. Ironically, my
statement that literals in parsers are confusing is based entirely upon
your stated premise.
Jim
More information about the antlr-interest
mailing list