[antlr-interest] Novice Question - Token for all characters from a given point to End of Line
antlr at mirality.co.nz
Tue Aug 5 14:00:18 PDT 2008
At 06:03 6/08/2008, Brisard, Fred D wrote:
>I have a keyword style grammar and have the need
>to accept all the characters until the end of
>line to be accepted as a single token.
>For example, I have a statement that is of the following type
>Command multiple arguments (EOL)
>Where Command can be a command name and the
>multiple arguments are one or more
>arguments. There can be from 1 to many
>arguments each argument does not have a fixed
>content it may be an integer, a string, a
>quoted string. The characters in the string can be most anything.
>I was looking for something similar to the
>multiple line comment technique using the
>greedy=false option. Collect all the characters
>following the Command into a single token.
Do you really *really* need everything to be one
token? I would have thought it'd be more useful
to lex each argument as a separate token (making
the best guess as to content type as is
possible), and then sort it out into command vs.
arguments at the parser level. (Of course, to do
that you'll need to make sure the EOL is not
hidden from the parser.)
More information about the antlr-interest