[antlr-interest] Semantic Predicates in a Lexer

Loring Craymer lgcraymer at yahoo.com
Fri Mar 20 11:53:33 PDT 2009






________________________________
From: Paul Bouché (NSN) <paul.bouche at nsn.com>
To: Jim Idle <jimi at temporal-wave.com>
Cc: antlr-interest at antlr.org
Sent: Friday, March 20, 2009 11:02:45 AM
Subject: Re: [antlr-interest] Semantic Predicates in a Lexer

...

Firstly, do not forget that you cannot set such a flag from the parser 
as the lexer runs first and creates all the tokens.
I know which I personally find a bug, if there was not a token
buffer, one could steer the lexer depending on parser state. This would
make the lexer much more powerfull. Ter stated in his book we need
tokens because this makes things easier and this is also how humans
analyse text, but humans don't buffer up all words in a page but read
them word by word and keep a short buffer of recently read tokens so as
to make context sensitive decisions ;-)

This is a misconception.  To allow feedback from parser to lexer, you need to be at a point in the parser grammar where k = 1.  This is difficult for the human to recognize; even when you insert feedback at such a point, subsequent modification of the grammar may increase the local value of k.  The use of such feedback between lex generated lexers and yacc generated parsers heavily depends on the fact that yacc only supports k=1.

--Loring


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090320/2bc8d485/attachment.html 


More information about the antlr-interest mailing list