[antlr-interest] unicode 16bit versus new 21bit stuff

[John D. Mitchell]

>>>>> "Terence" == Terence Parr <parrt at cs.usfca.edu> writes:
[...]

> I'm secretly planning to allow all sorts of cool stuff like parsers that
> can handle single char tokens w/o going to the lexer and so on.  Having
> the parser at runtime be able to distinguish char from token type just by
> looking at the value was going to be mighty handy.

What does that mean?  That there wouldn't be any, independent lexer?  Or
are you talking about some sort of coordination between the lexer and the
parser?


[...]

> What if we could encode the TokenStream channel in the token type so we
> can "tune" to only certain channels?

So is the idea would be to have multiple parsers, each tuning into different
channels or is it to have a single parser decide which set of channels it's
interested in or what?

> What if we had special tokens other than EOF?  Perhaps there will be a
> need.

Yes, if it was clean, I'd like to have a separate "meta" channel.  However,
if you're making the whole "channel" notion general, isn't this just e.g.,
channel zero?

Take care,
	John


 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/