[antlr-interest] Lexer Predicates?
Gavin Lambert
antlr at mirality.co.nz
Sat Aug 2 21:39:41 PDT 2008
At 11:06 3/08/2008, Foust wrote:
>Do lexer predicates work in v3?
That depends on what you mean. You can certainly use both
syntactic and semantic predicates within the lexer, but they can
only use lexer state.
Also, while I'm not entirely sure about this, I think predicates
in the lexer can only be used to decide between alts within a
single lexer rule. I vaguely recall some trouble when trying to
use them to decide between multiple lexer rules (at the top
level).
>I am used to yacc requesting tokens from the lexer as required.
>But ANTLR seems to run the lexer in its entirety and gather *all*
>the tokens first.
>
>So any predicates referencing states set in the Parser are
>effectively ignored.
>
>
>Is there any way around this?
No, that's just the way that ANTLR works. Lexing occurs entirely
separately, in a separate class, and has no access to nor
knowledge of parser state.
Generally speaking, you should keep your lexer fairly
straightforward and unambiguous, and defer semantic decisions (and
ambiguity resolution) until the parsing phase.
It's often easier to remember this separation if you write
separate lexer and parser grammars, instead of using the combined
grammar facility. (Or if you really want to use a combined
grammar, avoid using quoted literals within parser rules.)
More information about the antlr-interest
mailing list