[antlr-interest] Token position 0, -1

[Jim Idle]

Mike,

The 1 == 2 is necessary because that is the only way I could make the
conditional code work with the code generation techniques we have. In
different circumstances, you will see 1 == 1 and the code is incorporated.
The size of the C file only makes a difference with old debuggers that
used a 16 but line number field. The performance boost is significant.

Don't do all the rewriting stuff anyway, as it adds code and execution
time that you don't need. Just:

->^(NODE x t )

Then you want the start and end tokens from NODE in your tree grammar and
asking those tokens for their info will give you everything you want. In
general, do only the tree re-writing you need to make the tree unambiguous
- there is no need to move the text from one token in to another and so
on.

Jim

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Mike Lischke
> Sent: Tuesday, July 10, 2012 1:55 AM
> To: ANTLR Mailing List
> Subject: Re: [antlr-interest] Token position 0, -1
>
>
> Stefan, Jim,
>
> thanks for the hint about the start/stop tokens.
>
> >> Is there a similar construct if you have a rule instead of a token?
> >> For
> >> instance:
> >>
> >> target_table:
> >> 	qualified_identifier -> ^(TABLE_REF_ID qualified_identifier) ;
> >
> > I use in some case the start attribute of the subrule:
> >
> > target_table:
> > 	qualified_identifier ->
> ^(TABLE_REF_ID[$qualified_identifier.start]
> > qualified_identifier)
>
> This doesn't work with my grammar (using the C target). Instead of
> taking the location only the token text of my virtual token is set to
> NULL. The generated C code is:
>
>             if ( BACKTRACKING==0 )
>
>             {
>             	pANTLR3_REWRITE_RULE_SUBTREE_STREAM stream_retval;
>
>
> 	stream_retval=antlr3RewriteRuleSubtreeStreamNewAEE(ADAPTOR,
> RECOGNIZER, (pANTLR3_UINT8)"token retval", retval.tree != NULL ?
> retval.tree : NULL);
>
>             	root_0			    = (pANTLR3_BASE_TREE)(ADAPTOR-
> >nilNode(ADAPTOR));
>             	retval.tree    = root_0;
>             	// 580:23: -> ^(
> TABLE_REF_ID[$qualified_identifier.start] qualified_identifier )
>             	{
>             	    // MySQL55.g:580:26: ^(
> TABLE_REF_ID[$qualified_identifier.start] qualified_identifier )
>             	    {
>             	        pANTLR3_BASE_TREE root_1 =
> (pANTLR3_BASE_TREE)(ADAPTOR->nilNode(ADAPTOR));
>             	        root_1 = (pANTLR3_BASE_TREE)(ADAPTOR-
> >becomeRoot(ADAPTOR,
>
>             	        #if 1 == 2
>             	        	(pANTLR3_BASE_TREE)ADAPTOR-
> >createTypeTokenText(ADAPTOR, TABLE_REF_ID,
> TOKTEXT((qualified_identifier695.start)))
>             	        #else
>             	        	(pANTLR3_BASE_TREE)ADAPTOR-
> >createTypeText(ADAPTOR, TABLE_REF_ID,
> (pANTLR3_UINT8)(qualified_identifier695.start))
>             	        #endif
>
>             	        , root_1));
>
>             	        ADAPTOR->addChild(ADAPTOR, root_1,
> stream_qualified_identifier == NULL ? NULL :
> stream_qualified_identifier->nextTree(stream_qualified_identifier));
>
>             	        ADAPTOR->addChild(ADAPTOR, root_0, root_1);
>             	    }
>
>             	}
>
>
>             	retval.tree = root_0; // set result root
>
> which shows that the start token is used only to create the token text
> for root_1, which is clearly not what we want. Also, this 1 == 2 part
> seems quite useless and is just bloating the already huge file (almost
> 134K lines already).
>
> Mike
> --
> www.soft-gems.net
>
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address