[antlr-interest] Yet an another TokenStream ( for C++ Target )

[Gokulakannan Somasundaram]

Hi,
   I am planning to write a TokenStream with the following characteristics
a) It will fetch k number of tokens every time it is called. ( I am setting
a default of 100)
b) Every time a rule's action is executed in execution mode( when
backtracking == 0 ), it will delete all the tokens except the first and
last token.
c) After the tokens are deleted, if they are tried to be accessed, this
will throw an exception

Advantages:
Memory usage will be be optimal. It will be dependant on
a) amount of backtracking required
b) Maximum number of tokens covered in a single rule
c) Amount of look-ahead( This will never be a determining factor )

Disadvantages
a) As a rule will lose its tokens except start and stop token, any attempt
to refer to those tokens would result in a exception.

But this can be overcome by storing the required data of those tokens in a
variable storage, during the execution of the rule.

This can be enabled by changing the trait TOKENS_ACCESSED_FROM_OWNING_RULE
to false.

Any Comments / Suggestions?

Thanks,
Gokul.