[antlr-interest] Lexer Predicates?

Gavin Lambert antlr at mirality.co.nz
Sat Aug 2 21:39:41 PDT 2008


At 11:06 3/08/2008, Foust wrote:
>Do lexer predicates work in v3?

That depends on what you mean.  You can certainly use both 
syntactic and semantic predicates within the lexer, but they can 
only use lexer state.

Also, while I'm not entirely sure about this, I think predicates 
in the lexer can only be used to decide between alts within a 
single lexer rule.  I vaguely recall some trouble when trying to 
use them to decide between multiple lexer rules (at the top 
level).

>I am used to yacc requesting tokens from the lexer as required. 
>But ANTLR seems to run the lexer in its entirety and gather *all* 
>the tokens first.
>
>So any predicates referencing states set in the Parser are 
>effectively ignored.
> 
>
>Is there any way around this?

No, that's just the way that ANTLR works.  Lexing occurs entirely 
separately, in a separate class, and has no access to nor 
knowledge of parser state.

Generally speaking, you should keep your lexer fairly 
straightforward and unambiguous, and defer semantic decisions (and 
ambiguity resolution) until the parsing phase.

It's often easier to remember this separation if you write 
separate lexer and parser grammars, instead of using the combined 
grammar facility.  (Or if you really want to use a combined 
grammar, avoid using quoted literals within parser rules.)



More information about the antlr-interest mailing list