[antlr-interest] unicode 16bit versus new 21bit stuff

[Ric Klaren]

On Sat, Jun 19, 2004 at 10:09:34AM -0700, Terence Parr wrote:
> I'm secretly planning to allow all sorts of cool stuff like parsers 
> that can handle single char tokens w/o going to the lexer and so on.  

I'm completely missing your point here I guess...

> Having the parser at runtime be able to distinguish char from token 
> type just by looking at the value was going to be mighty handy.

Are you making the line between lexer and parser fuzzier?  
How about (LT(1).getText().length() == 1) ? Again I think I'm missing your
point.

> In the previous version, I made a number of decisions based upon the 
> current state of the art in CPU speed / architecture, which of course 
> changes pretty damn fast.  I wonder if we shouldn't just go 64 bit for 
> the token types leaving a full 32-bits for characters and for token 
> types all within the same value. 

Again what is the gain from this ? Will a struct/class not be cleaner at a
minor expense in speed and gain in portability/extendability/code
readability? (Let me state again I don't care much about the analysis
engine but I do care about the support lib)

> Who knows, this may also make some other things convenient.  What if we
> could encode the TokenStream channel in the token type so we can "tune" to
> only certain channels? 

This could be something usefull. It would complement nicely with
TokenStream multiplexing is my guess. Although you'd be demultiplexing
things with the channel. Then you'd be building a tree in one parser but
how would the stuff from the other parser end up in the tree (if you'd want
it there)

> What if we had special tokens other than EOF?
> Perhaps there will be a need.

Advice from a notorious overengineerder (me): don't! ;) KISS is the word
especially for a first version.

> Hmm...I wonder how fast 64-bit processors will become the norm (G5s are 
> there and AMD is too, right?)?  How horrible does Java do 64-bit ints 
> now for comparison and other rot?  ANTLR 3.0 won't be available for a 
> while...perhaps 64 bits ain't that bad an idea.  BTW, passing around 
> another variable is unwieldly coding wise and will just be two 32-bit 
> ints anyway, right?

Why not pass a struct/class ? Again I'd be carefull moving into the 64 bit
arena too soon. Wait till the bugs die out first. This is probably a bigger
issue on the C++ side than on the java side (more vendors more problems).
 
Cheers,

Ric
-- 
-----+++++*****************************************************+++++++++-------
    ---- Ric Klaren ----- j.klaren at utwente.nl ----- +31 53 4893755  ----
-----+++++*****************************************************+++++++++-------
  There are 10 kinds of people - those that understand binary and
  those that don't. --- Unknown



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/