[antlr-interest] Yet an another TokenStream ( for C++ Target )

[Gokulakannan Somasundaram]

one small correction.

This can be enabled by changing the trait TOKENS_ACCESSED_FROM_OWNING_RULE
to true.

Thanks,
Gokul.

On Sun, Feb 19, 2012 at 1:23 AM, Gokulakannan Somasundaram <
gokul007 at gmail.com> wrote:

> Hi,
>    I am planning to write a TokenStream with the following characteristics
> a) It will fetch k number of tokens every time it is called. ( I am
> setting a default of 100)
> b) Every time a rule's action is executed in execution mode( when
> backtracking == 0 ), it will delete all the tokens except the first and
> last token.
> c) After the tokens are deleted, if they are tried to be accessed, this
> will throw an exception
>
> Advantages:
> Memory usage will be be optimal. It will be dependant on
> a) amount of backtracking required
> b) Maximum number of tokens covered in a single rule
> c) Amount of look-ahead( This will never be a determining factor )
>
> Disadvantages
> a) As a rule will lose its tokens except start and stop token, any attempt
> to refer to those tokens would result in a exception.
>
> But this can be overcome by storing the required data of those tokens in a
> variable storage, during the execution of the rule.
>
> This can be enabled by changing the trait TOKENS_ACCESSED_FROM_OWNING_RULE
> to false.
>
> Any Comments / Suggestions?
>
> Thanks,
> Gokul.
>