[antlr-interest] Tokens vs. Characters in Lexer/MismatchedTokenException
rmann at latencyzero.com
Sun Jan 18 13:30:35 PST 2009
On Jan 18, 2009, at 13:12:30, Terence Parr wrote:
> ah. Yes, I get it now. Take of a lexer as a parser that parses
> characters instead of tokens. In this way I have generalized the
> notion of a recognizer so that we represent any element in the stream
> as an integer vocabulary symbol "type".
> sorry for the confusion.
No worries. The use of the Java primitive type "int" threw me a bit,
is all. I've typedef'd a "CharTokenT" to use, might find a better name
later. I guess this is okay because you'll only ever be looking at
characters or token types exclusively, never a mix.
More information about the antlr-interest