[antlr-interest] "Tokens"
Steve Bennett
stevagewp at gmail.com
Sun Nov 18 07:49:17 PST 2007
On 11/18/07, Johannes Luber <jaluber at gmx.de> wrote:
> '!' won't be translated to an EXCLAMATION, but to e.g. Token23, so you
> have then a MismatchException for r or for l, depending on the priority
> between EXCLAMATION and Token23. Personally I would use '!' instead
> EXCLAMATION, as it seems clearer to me. In the case of separated parsers
> you have to use the EXCLAMATION-style, as ANTLR won't allow literals in
> pure parser or tree parser grammars. It would have to generate a second
> lexer in that case and as only one lexer is used to split the input
> stream, the second wouldn't be ever executed.
Thanks for that explanation. I guess it would probably be preferable
to do something like this:
minus: '-';
hyphen: '-';
so that you can refer to the same token by either name without
stuffing anything else up.
Incidentally, is there a way to visualise the tokenisation in
ANTLRWorks? I seem to have so many problems with tokens, it would be
great to just see the definitive list of how everything has been
broken up. I guess it shouldn't be too hard for me to just add a dump
from the TokenStream in my java test rig, too...
Steve (who's spent the whole day struggling with some lexing issues
before finally stumbling on a way of using a semantic predicate with
this code: boolean textis(String mw) { return
input.LT(1).getText().equalsIgnoreCase(mw); } )
More information about the antlr-interest
mailing list