[antlr-interest] unicode 16bit versus new 21bit stuff
Terence Parr
parrt at cs.usfca.edu
Sun Jun 20 11:58:36 PDT 2004
On Jun 20, 2004, at 11:49 AM, John D. Mitchell wrote:
>>>>>> "Terence" == Terence Parr <parrt at cs.usfca.edu> writes:
> [...]
>
>> I'm secretly planning to allow all sorts of cool stuff like parsers
>> that
>> can handle single char tokens w/o going to the lexer and so on.
>> Having
>> the parser at runtime be able to distinguish char from token type
>> just by
>> looking at the value was going to be mighty handy.
>
> What does that mean? That there wouldn't be any, independent lexer?
> Or
> are you talking about some sort of coordination between the lexer and
> the
> parser?
I'm not sure...my intuition says there is something there. For sure,
I'm ultimately going to have a context-sensitive lexer where it is
goal-oriented unlike existing lexers. I.e., if your parser can match
either an INT or a SEMI at a spot, it asks the lexer to match one of
those two only--not the entire possible set.
>> What if we could encode the TokenStream channel in the token type so
>> we
>> can "tune" to only certain channels?
>
> So is the idea would be to have multiple parsers, each tuning into
> different
> channels or is it to have a single parser decide which set of channels
> it's
> interested in or what?
Mostly the latter, but could be both. Could have a parser for javadoc
and a parser for java code running in parallel :)
>> What if we had special tokens other than EOF? Perhaps there will be a
>> need.
>
> Yes, if it was clean, I'd like to have a separate "meta" channel.
> However,
> if you're making the whole "channel" notion general, isn't this just
> e.g.,
> channel zero?
Yup...the channel is probably best handled as it is now with
TokenStream, but ya never know :)
Ter
--
CS Professor & Grad Director, University of San Francisco
Creator, ANTLR Parser Generator, http://www.antlr.org
Cofounder, http://www.jguru.com
Cofounder, http://www.knowspam.net enjoy email again!
Cofounder, http://www.peerscope.com pure link sharing
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/antlr-interest/
<*> To unsubscribe from this group, send an email to:
antlr-interest-unsubscribe at yahoogroups.com
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
More information about the antlr-interest
mailing list