[antlr-interest] Novice Question - Token for all characters from a given point to End of Line

[Gavin Lambert]

At 06:03 6/08/2008, Brisard, Fred D wrote:

>I have a keyword style grammar and have the need 
>to accept all the characters until the end of 
>line to be accepted as a single token.
>
>For example, I have a statement that is of the following type
>
>Command multiple arguments (EOL)
>
>Where Command can be a command name and the 
>multiple arguments are one or more 
>arguments.  There can be from 1 to many 
>arguments – each argument does not have a fixed 
>content – it may be an integer, a string, a 
>quoted string.  The characters in the string can be most anything.
>
>I was looking for something similar to the 
>multiple line comment technique using the 
>greedy=false option.  Collect all the characters 
>following the Command into a single token.

Do you really *really* need everything to be one 
token?  I would have thought it'd be more useful 
to lex each argument as a separate token (making 
the best guess as to content type as is 
possible), and then sort it out into command vs. 
arguments at the parser level.  (Of course, to do 
that you'll need to make sure the EOL is not 
hidden from the parser.)