[antlr-interest] Context-sensitive lexing
shmuel siegel
antlr at shmuelhome.mine.nu
Mon Nov 19 00:48:28 PST 2007
Steve Bennett wrote:
> .....
> I gather that most programming languages don't have this drama,
> because there are generally two lexing situations: normal text, where
> { and -> are special tokens, or strings/comments, where /* blah -> {
> blah */ is treated as a single token. But what would you do if you
> wanted to actually parse the contents of that comment, rather than
> making it a monolithic token?
> .......
> Steve
>
For cases like comments, there is an alternative to island grammars.
Since the lexer is capable of recognizing the boundaries of the
comment, you can have it return a comment to the parser. The parser
calls another lexerer/parser passing them the content of the comment.
This involves double lexing but should be fast enough.
Shmuel
More information about the antlr-interest
mailing list