[antlr-interest] Yet an another TokenStream ( for C++ Target )

Gavin Lambert antlr at mirality.co.nz
Wed Feb 22 00:01:49 PST 2012

At 09:12 19/02/2012, Gokulakannan Somasundaram wrote:
 >>> b) Every time a rule's action is executed in execution mode(
 >>> when backtracking == 0 ), it will delete all the tokens 
 >>> the first and last token.
 >If a user attempts to look ahead, there won't be any issue. If a 

 >user looks backward, it might result i an exception. If the user 

 >wants to access any token, it has to be attempted only from
 >within the rule. This is the discipline a parser programmer has
 >to follow to reduce the memory footprint. After the rule's scope 

 >is completed, the tokens might get deleted. So, the necessary
 >info has to be stored in some variable at rule scope / global
 >scope and accessed from there.

Why not have an option that wraps all Token references inside a 
std::shared_ptr (or std::tr1::shared_ptr or boost::shared_ptr for 
backwards compatibility)?  That way the action code can cache 
particular tokens for as long as required, while uninteresting 
tokens (even start/stop tokens if the action code clears them) can 
get automatically deleted as they fall out of scope.

(It shouldn't be the default though, as shared_ptrs aren't cheap.)

Gavin Lambert, Mirality Systems
Anyone who goes to a psychiatrist ought to have their head 

Few women admit their age.  Few men act theirs.

More information about the antlr-interest mailing list