[antlr-interest] Yet an another TokenStream ( for C++ Target )

Gokulakannan Somasundaram gokul007 at gmail.com
Sat Feb 18 12:12:35 PST 2012


>
>
>   I played with idea myself, since it is the most efficient way, but I
> don't see how it can work for any LL(*) grammar without adding grammar
> constraints. If k is set in the options then it is much easier and actually
> someone did this a few years ago:
> http://markmail.org/message/ej25vkco44ppkaxt
>
>
> That was definitely a good attempt. But it has a constant size token list.
So when we write a rule with more  than the stored tokens and try to access
all the tokens ( say for getting the text ), then it is not possible. This
is a problem, which cannot be solved in this approach.


>
> b) Every time a rule's action is executed in execution mode( when
>> backtracking == 0 ), it will delete all the tokens except the first and
>> last token.
>>
> I don't quite follow you here. Actions can appear anywhere in rules and
> since you'll be in a rule at all times, the only way to know which tokens
> are safe to delete is to track all the tokens used by rule actions and only
> delete them after the action executes.
>

We are talking only about named rules here.


>   The codegen probably knows about all calls to $token_vars in actions but
> it likely doesn't know about calls to LT(n). I think this approach can work
> but it will probably require that any user code cannot access the token
> stream or any tokens directly except with $ references.
>
> No i am not attempting to knows about all calls to $token_vars in actions.
If a user attempts to look ahead, there won't be any issue. If a user looks
backward, it might result i an exception. If the user wants to access any
token, it has to be attempted only from within the rule. This is the
discipline a parser programmer has to follow to reduce the memory
footprint. After the rule's scope is completed, the tokens might get
deleted. So, the necessary info has to be stored in some variable at rule
scope / global scope and accessed from there.

Gokul.


More information about the antlr-interest mailing list