[antlr-interest] Token position 0, -1

Mike Lischke mike at lischke-online.de
Tue Jul 10 01:55:09 PDT 2012


Stefan, Jim,

thanks for the hint about the start/stop tokens.

>> Is there a similar construct if you have a rule instead of a token? For
>> instance:
>> 
>> target_table:
>> 	qualified_identifier -> ^(TABLE_REF_ID qualified_identifier)
>> ;
> 
> I use in some case the start attribute of the subrule:
> 
> target_table:
> 	qualified_identifier -> ^(TABLE_REF_ID[$qualified_identifier.start] 
> qualified_identifier)

This doesn't work with my grammar (using the C target). Instead of taking the location only the token text of my virtual token is set to NULL. The generated C code is:

            if ( BACKTRACKING==0 ) 

            {
            	pANTLR3_REWRITE_RULE_SUBTREE_STREAM stream_retval;

            	stream_retval=antlr3RewriteRuleSubtreeStreamNewAEE(ADAPTOR,  RECOGNIZER, (pANTLR3_UINT8)"token retval", retval.tree != NULL ? retval.tree : NULL);

            	root_0			    = (pANTLR3_BASE_TREE)(ADAPTOR->nilNode(ADAPTOR));
            	retval.tree    = root_0;
            	// 580:23: -> ^( TABLE_REF_ID[$qualified_identifier.start] qualified_identifier )
            	{
            	    // MySQL55.g:580:26: ^( TABLE_REF_ID[$qualified_identifier.start] qualified_identifier )
            	    {
            	        pANTLR3_BASE_TREE root_1 = (pANTLR3_BASE_TREE)(ADAPTOR->nilNode(ADAPTOR));
            	        root_1 = (pANTLR3_BASE_TREE)(ADAPTOR->becomeRoot(ADAPTOR, 

            	        #if 1 == 2
            	        	(pANTLR3_BASE_TREE)ADAPTOR->createTypeTokenText(ADAPTOR, TABLE_REF_ID, TOKTEXT((qualified_identifier695.start)))
            	        #else
            	        	(pANTLR3_BASE_TREE)ADAPTOR->createTypeText(ADAPTOR, TABLE_REF_ID, (pANTLR3_UINT8)(qualified_identifier695.start))
            	        #endif

            	        , root_1));

            	        ADAPTOR->addChild(ADAPTOR, root_1, stream_qualified_identifier == NULL ? NULL : stream_qualified_identifier->nextTree(stream_qualified_identifier));

            	        ADAPTOR->addChild(ADAPTOR, root_0, root_1);
            	    }

            	}


            	retval.tree = root_0; // set result root

which shows that the start token is used only to create the token text for root_1, which is clearly not what we want. Also, this 1 == 2 part seems quite useless and is just bloating the already huge file (almost 134K lines already).

Mike
-- 
www.soft-gems.net




More information about the antlr-interest mailing list