[antlr-interest] unicode 16bit versus new 21bit stuff

[Terence Parr]

On Jun 20, 2004, at 11:49 AM, John D. Mitchell wrote:

>>>>>> "Terence" == Terence Parr <parrt at cs.usfca.edu> writes:
> [...]
>
>> I'm secretly planning to allow all sorts of cool stuff like parsers 
>> that
>> can handle single char tokens w/o going to the lexer and so on.  
>> Having
>> the parser at runtime be able to distinguish char from token type 
>> just by
>> looking at the value was going to be mighty handy.
>
> What does that mean?  That there wouldn't be any, independent lexer?  
> Or
> are you talking about some sort of coordination between the lexer and 
> the
> parser?

I'm not sure...my intuition says there is something there.  For sure, 
I'm ultimately going to have a context-sensitive lexer where it is 
goal-oriented unlike existing lexers.  I.e., if your parser can match 
either an INT or a SEMI at a spot, it asks the lexer to match one of 
those two only--not the entire possible set.

>> What if we could encode the TokenStream channel in the token type so 
>> we
>> can "tune" to only certain channels?
>
> So is the idea would be to have multiple parsers, each tuning into 
> different
> channels or is it to have a single parser decide which set of channels 
> it's
> interested in or what?

Mostly the latter, but could be both.  Could have a parser for javadoc 
and a parser for java code running in parallel :)

>> What if we had special tokens other than EOF?  Perhaps there will be a
>> need.
>
> Yes, if it was clean, I'd like to have a separate "meta" channel.  
> However,
> if you're making the whole "channel" notion general, isn't this just 
> e.g.,
> channel zero?

Yup...the channel is probably best handled as it is now with 
TokenStream, but ya never know :)

Ter
--
CS Professor & Grad Director, University of San Francisco
Creator, ANTLR Parser Generator, http://www.antlr.org
Cofounder, http://www.jguru.com
Cofounder, http://www.knowspam.net enjoy email again!
Cofounder, http://www.peerscope.com pure link sharing





 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/