[antlr-interest] TokenStreamRewriteEngine and C++ and RefCounters

[Ric Klaren]

Hi,

The RefCounter stuff kept nagging in the back of my mind so I tinkered with
it a bit earlier as expected.

On Tue, Apr 20, 2004 at 09:13:18PM +0700, Andrey R. Urazov wrote:
> Is it possible to release some snapshot sooner? If not, could you
> possibly pinpoint what's the problem with the RefToken since I'll need
> the functionality of a rewrite engine within two weeks and will have to
> implement it on my own if it's not yet available by the time.

I got the TokenStreamRewriter going mostly it seems (compared to the java
version I now only see a small difference in whitespace handling probably
something small somewhere).

> And also, out of curiosity, what's so sucky about these refcounters in
> general?

Two extra for the list:
Antlr's refcounter is non-intrusive.
- Because how it's set up it's slower as Mark's for instance (one extra
  null pointer check per smartpointer dereference)
- I'm not sure but I think Mark's should handle covariant return values
  because it's intrusive.
  (e.g. having the factories return RefToSubclass in stead of RefToken as I
  have to with Antlr's refcounter)

On Wed, Apr 21, 2004 at 12:42:47PM -0700, Mark Lentczner wrote:
> > Although I got a nice hunch from looking at your code :)
> Three cheers for symbiotic open source.

It looks like the hunch was good. I reimplemented RefCount into a
TokenRefCount that does the right thing and a bit more.

Basically the problem was that there was not one base class for all the
reference counted objects together with no way to convert the ref's from
one to another (mostly due to class layout).

Problem of the new implementation is that it's much too permissive in
conversions (guess I'm for extremes) it would merrily convert
TokenWithIndex to RefTokenWithIndex (because Token is used as the
underlying pointer)

I fixed that by adding a bit of Loki's template magic which automagically
adds an assert and dynamic cast to the conversion (could be a policy object
too I guess) if it's needed. Works quite nice I may say.

If there's interest I can provide a package for people to look at.

I also looked at replacing antlr's refcounters by Mark's implemtation which
worked also flawlessly (did not check it for too permissive conversions
though, and it needed a few small tweaks) Using Marks's intrusive reference
counter might give a nice speedup for treewalkers.

> While I am a major fan of garbage collection (I was on the Smalltalk
> team at Apple, and have worked on several other garbage collectors
> after that), it would be a show-stopper for our use of Antlr:  Our
> environment is memory sensitive.

With respect to this I was thinking of doing garbage collection directly
after building tree parts at moments where it would be easy (I hope) to see
what is in the generated tree and what not.

Another question for Mark: if we were to take the countable classes from
Wheat, would you be willing to put them under the same license as antlr
e.g. public domain (or a BSD style license as planned for antlr 3) (just
curious as it would safe some license compatibility headaches ;) )?

Cheers,

Ric
--
-----+++++*****************************************************+++++++++-------
    ---- Ric Klaren ----- j.klaren at utwente.nl ----- +31 53 4893755  ----
-----+++++*****************************************************+++++++++-------
  "Of all the things I've lost I miss my mind the most --- Ozzy Osbourne



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/