[antlr-interest] Semantic Predicates in a Lexer
Gavin Lambert
antlr at mirality.co.nz
Fri Mar 20 13:01:03 PDT 2009
At 07:29 21/03/2009, Sam Barnett-Cormack wrote:
>as I work on the additional bits of ASN.1 I'm finding reasons
>to really want to know what the last 2 tokens generated were,
>and use those for gating in the lexer. If I could do that
>easily (i.e. without adding an action to every single lexer
>rule), it'd make my life easier. I suppose a lexer subclass
>could do it - override the emit stuff and add functions to
>access already-generated tokens.
Well, if you were in the parser you could access
previously-consumed tokens with LA(-1), LA(-2), etc. You can do
the same thing in the lexer but it'll give you the characters
instead of the tokens.
It'd be hard to write that sort of code anyway -- you can't just
look at "the single previously generated token" because there's a
very high chance that it's a comment or whitespace or something,
meaning you'd have to look further back, and it just quickly gets
unwieldy.
Probably a better idea is to follow the example in the wiki about
generating multiple tokens from a single lexer rule. That way you
can keep everything under your own control and only emit the set
once you're sure that's what you really want :)
More information about the antlr-interest
mailing list