[antlr-interest] MismatchedTokenException and how to find errors in ANTLRWorks

[Jim Idle]

> -----Original Message-----
> From: Guntis Ozols [mailto:guntiso at latnet.lv]


> > 2. It's too easy to forget that each quoted string produces a
> > lexer rule (which may conflict with other rules), making it harder
> > to find the source of ambiguity problems.  In addition, "'x' 'y'"
> > is not the same as "'xy'" in the parser, though it is in the
> > lexer; this can cause confusion.
> 
> Sure it is confusing. I fell into this 'xy' trap, too.
> If COLON is defined and STAR is defined, then creating
> another token for ':*' in a *parser* rule is confusing.
> It should be fixed in antlr.

What should be fixed? 'x' 'y' IS something different to the parser than 
the lexer! ':' '*' and ':*' ARE different tokens.

> At least, antlr should emit 'anonymous token' warning[s].
> 

This post is a classic example of the problem. I am not trying to single 
out the poster, just point out the fact that the literals in the parser 
create great confusion when your knowledge of the tool is low, and I 
don't think you should use this until you have a better command of the 
tool. One certainly should not ask for things to be 'fixed'

There isn't anything wrong with the tool, it is doing what you tell it 
to do. The token isnt anonymous, it is called T66 or whatever, the 
runtime doesn't know that you created it by specifying it in the parser. 
So, I don't see why "anonymous token" would help you at all.

> I prefer literals because of simplicity / readability.

However, you are showing in fact that if anything, it is more 
complicated, not simpler. Readability is subjective. Is ':' anymore 
readable than COLON once you have the simple rule that words starting 
with Upper Case are tokens from the lexer?

> The tool needs to be patched, not users.

A tool should strive to be as simple as possible, but no simpler :-) 
While that statement makes sense, you are an under an obligation to 
yourself to try to make sure you understand the tool too. Ironically, my 
statement that literals in parsers are confusing is based entirely upon 
your stated premise.

Jim