[antlr-interest] unquoted strings in small line-based language

[Gavin Lambert]

At 16:42 18/10/2007, John Pye wrote:
 >How can I make the lexer *not* peer into this free-text and 
attempt
 >to identify tokens in it? How would I then catch the resulting
 >string in my parser?

The only way to do this is to do all the work in the 
lexer.  Remember that lexing is a completely independent step that 
occurs before parsing, so the parser cannot "inform" the lexer 
about any context.

Something like this would probably do the trick:

CASE
   :  'CASE' WSChar+ INTEGER WSChar+ (~'\n')*
   ;

fragment WSChar : ' ' | '\t';

WS : WSChar { $channel = HIDDEN; };

This will match the whole CASE line as a single token, which isn't 
ideal.  You could probably improve this by using the technique 
discussed on the Wiki for making a lexer rule output more than one 
token.