[antlr-interest] Semantic Predicates in a Lexer

[Sam Barnett-Cormack]

Paul Bouché (NSN) wrote:
> Hi Jim,
> 
> thanks for your response.
> 
> Jim Idle schrieb:
>> Paul Bouché (NSN) wrote:
>>> Hi,
>> Firstly, do not forget that you cannot set such a flag from the parser 
>> as the lexer runs first and creates all the tokens.
> I know which I personally find a bug, if there was not a token buffer, 
> one could steer the lexer depending on parser state. This would make the 
> lexer much more powerfull. Ter stated in his book we need tokens because 
> this makes things easier and this is also how humans analyse text, but 
> humans don't buffer up all words in a page but read them word by word 
> and keep a short buffer of recently read tokens so as to make context 
> sensitive decisions ;-)

Well, simultaneously lexing and parsing would be awkward for many, many 
reasons, but as I work on the additional bits of ASN.1 I'm finding 
reasons to really want to know what the last 2 tokens generated were, 
and use those for gating in the lexer. If I could do that easily (i.e. 
without adding an action to every single lexer rule), it'd make my life 
easier. I suppose a lexer subclass could do it - override the emit stuff 
and add functions to access already-generated tokens.

-- 
Sam Barnett-Cormack