[antlr-interest] Writing (for now) just a lexer

[Evan Driscoll]

Evan Driscoll wrote:
> Thanks for the responses before. I mention them below. I have a new
> question though, which is that it doesn't seem to ignore
> hidden/whitespace tokens.

Sorry to reply to myself, but I should be a little more explicit here.

If I run the program whose code I linked to before (which basically just
calls nextToken() for a while and echoes each thing it gets), if I give
it the input "10 443.7" followed by a newline, I will get this as the
output:

  1:0: INT_LITERAL
  org.antlr.runtime.CommonToken
  1:3: FLOAT_LITERAL
  org.antlr.runtime.CommonToken
  1:8: NEWLINE                        <--- note the newline
  org.antlr.runtime.CommonToken
  0:-1: EOF
  org.antlr.runtime.CommonToken



> Does the $channel=HIDDEN only work if you start using it in the context
> of a lexer or something?
       ^^^^^
This should say "parser, not lexer. (In other words, hidden tokens will
be ignored by parser rules that don't explicitly ask for them, but since
there's an implicit nextToken parser rule for lexer grammars that looks
like 'nextToken: INT_LITERAL | FLOAT_LITERAL | WHITESPACE | NEWLINE' for
me, and it explicitly lists the newline, it's returned?)


Finally, running skip() instead of $channel=HIDDEN does do what I want,
so this isn't a critical issue either.

Evan Driscoll