[antlr-interest] Re: trees with payloads??

[John D. Mitchell]

>>>>> "lgcraymer" == lgcraymer  <lgc at mail1.jpl.nasa.gov> writes:
>>>>>> In antlr-interest at yahoogroups.com, "micheal_jor" <open.zone at v...>
[...]

>> Hmm, I am more inclined to eliminate [the cost of] Token objects
>> [almost] entirely. I say almost because a mechanism is still needed to
>> transfer attributes like lexeme/line/col info to AST node objects.

In general, I don't think that's possible.  I.e., any "solution" you come
up with is just moving around the time/space tradeoffs to different parts.

As Ter noted last week, it's not really the "column" but rather it's an
character index relative to the current line.

[...]
> Token management in lexers is an art.  You have to avoid copying data as
> much as possible for performance reasons and minimize memory allocation.
> llk does a reasonably good job; I think that Chris Leung took advantage
> of lessons learned from flex and other lexer generators.  Performance
> paranoia helps.  Payload objects could help here, too--you can keep text
> start and end pointers into a memory-mapped text file instead of
> allocating and copying strings.

The current Antlr v3 prototype keeps indices into the data buffer of the
start and end of the token.

Take care,
	John


 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/