[antlr-interest] Re: Token Streams, multiple lexers - how doe s lexer access selector?

[mzukowski at yci.com]

In particular the lexer is always at least k tokens ahead of the parser.  So
instead of telling the lexer "I see a Y so switch lexers on the next token I
see," you are saying "I see a Y so switch lexing on your next token which is
at least k but possibly many more tokens ahead of what I just matched."
There is a chance that the parser hasn't requested a new token from the
lexer yet, so buffering may give the illusion that the lexer is less than k
tokens ahead when it is told to switch.  But with syntactic predicates you
could conceptually lex the entire file before you actually commit to parsing
a particular alternative.  That's why the call it "infinite lookahead."

Monty

-----Original Message-----
From: Ric Klaren [mailto:klaren at cs.utwente.nl]
Sent: Thursday, December 19, 2002 7:22 AM
To: antlr-interest at yahoogroups.com
Subject: Re: [antlr-interest] Re: Token Streams, multiple lexers - how
does lexer access selector?


Hi,

On Thu, Dec 19, 2002 at 03:15:49PM -0000, davidjpenton2002
<djplists at arrowsash.com> wrote:
> Is it "safe" to push and pop lexers using a selector object (as in
> the example in the ref manual) from within a parser action instead of 
> from within a lexer rule?

It's unsafe to do from parser. In some situations you might get away with
it, but it's something that will break very easily. Think guns and feet
with holes in them.

Ric
-- 
-----+++++*****************************************************+++++++++----
---
    ---- Ric Klaren ----- klaren at cs.utwente.nl ----- +31 53 4893722  ----
-----+++++*****************************************************+++++++++----
---
 "Don't call me stupid." "Oh, right. To call you stupid would be an insult
    to stupid people. I've known sheep that could outwit you! I've worn
              dresses with higher IQs!" --- A Fish Called Wanda


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/