[antlr-interest] Strange ANTLR behavior when using heterogeneous ASTs

[Andrey R. Urazov]

Hello,

I found it very strange that when in the tokens section I define my own
AST types for tokens, it is NOT a direct instruction to generate code
performing custom ASTs' factory methods registration in the
`initializeASTfactory' member function of the generated parser class.
The factory method gets registered only when the token is used somewhere
in the grammar but not in a grammar action.

For example, for the following grammar file:

------------------------------------------

options { language = Cpp; }

class TestParser extends Parser;

options { buildAST = true; }

tokens {
    ACTION_TOKEN<AST=MyAST>;
    GRAMMAR_TOKEN<AST=MyAST>;
}

start!
:
    GRAMMAR_TOKEN
    {
        #start = #[ACTION_TOKEN];
    }
;

------------------------------------------

ANTLR generates the following:

------------------------------------------

...

void TestParser::initializeASTFactory( ANTLR_USE_NAMESPACE(antlr)ASTFactory& factory )
{
	factory.registerFactory(5, "MyAST", MyAST::factory);
	factory.setMaxNodeType(5);
}
const char* TestParser::tokenNames[] = {
	"<0>",
	"EOF",
	"<2>",
	"NULL_TREE_LOOKAHEAD",
	"ACTION_TOKEN",
	"GRAMMAR_TOKEN",
	0
};

------------------------------------------

I.e. `registerFactory' was only called for GRAMMAR_TOKEN explicitly
named in the grammar but not for the ACTION_TOKEN which is used in an
action.

I don't know whether this is a bug or done intentionally, but to me it
seems very strange. Maybe there was some motivation to do this in order
to, for some reason, demarcate usual and imaginary tokens. But I don't
understand this. To me, it's natural to want to fix AST types for tokens
--- no matter whether they are real or imaginary --- at once and then
operate on them without explicit AST type specification.


Best regards,
  Andrey Urazov



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/