[antlr-interest] Context-sensitive lexing

[Gavin Lambert]

At 21:48 19/11/2007, shmuel siegel wrote:
 >Since the lexer is capable of recognizing  the boundaries of
 >the comment, you can have it return a comment to the parser.
 >The parser calls another lexerer/parser passing them the
 >content of the comment.
 >This involves double lexing but should be fast enough.

Actually you can do that even for constructs that can only be 
recognised by the parser too.  Each token carries with it a 
location in the input stream, so if you can find two boundary 
tokens then you can extract the substream between them and pass it 
to another lexer/parser combo, if you want.