[antlr-interest] "Tokens"

[Steve Bennett]

On 11/18/07, Johannes Luber <jaluber at gmx.de> wrote:
> '!' won't be translated to an EXCLAMATION, but to e.g. Token23, so you
> have then a MismatchException for r or for l, depending on the priority
> between EXCLAMATION and Token23. Personally I would use '!' instead
> EXCLAMATION, as it seems clearer to me. In the case of separated parsers
> you have to use the EXCLAMATION-style, as ANTLR won't allow literals in
> pure parser or tree parser grammars. It would have to generate a second
> lexer in that case and as only one lexer is used to split the input
> stream, the second wouldn't be ever executed.

Thanks for that explanation. I guess it would probably be preferable
to do something like this:

minus: '-';
hyphen: '-';

so that you can refer to the same token by either name without
stuffing anything else up.

Incidentally, is there a way to visualise the tokenisation in
ANTLRWorks? I seem to have so many problems with tokens, it would be
great to just see the definitive list of how everything has been
broken up. I guess it shouldn't be too hard for me to just add a dump
from the TokenStream in my java test rig, too...

Steve (who's spent the whole day struggling with some lexing issues
before finally stumbling on a way of using a semantic predicate with
this code:   boolean textis(String mw) {  return
input.LT(1).getText().equalsIgnoreCase(mw);  } )