[antlr-interest] More on TokenWithIndex

[Paul J. Lucas]

	I've been playing with this stuff for several days now and have
	something that works well.

	I modified Terence's ASTMinMax class and added a Visitor
	interface.

	To solve the left-factored problem, I do:

		declareThing
		    : d:DECLARE! { markTokenIndex( d ); }
		      ( functionDecl
		      | varDecl
		      | // ...
		      )
		    ;

	where markTokenIndex() simply sets a parser class variable that
	can be obtained later; then:

		varDecl
		    : v:VARIABLE^ varRefTypeDecl ASSIGN! expr semi:SEMICOLON!
		    	{
			    #v.setType( VAR_DECL );
			    #v.setText( "VAR_DECL" );
			    setMinMaxIndex( #v, semi );
			}
		    ;

	where the two-argument form of setMinMaxIndex() sets the min/max
	indicies of #v to the previously-marked token and the semicolon.
	There are many overloaded variations of setMinMaxIndex(): some
	taken tokens as arguments, some take AST nodes, and combinations
	of both.

	Most rules don't need any call to setMinMaxIndex() at all.  The
	only ones that do are for left-factored rules as shown and also
	for those where the max index is a non-included terminal, e.g.,:

		parenthesizedExpr
		    : lp:LPAREN! expr rp:RPAREN!
		    	{
			    setMinMaxIndex( ##, lp, rp );
			}
		    ;
	or:
		functionCall
		    : f:FUNCTION_NAME LPAREN! (exprList)? rp:RPAREN!
		    	{
		            #f.setType( FUNCTION_CALL );
		            #f.setText( "FUNCTION_CALL" );
		            setMinMaxIndex( #f, f, rp );
			}
		    ;

	Note that "#f" refers to the AST node and 'f' refers to the
	token.

	To set all the min/max indicies for the entire tree once it's
	completely built, I have a visitor that post-order traverses
	it.  As part of the initialize() method on my derived AST
	class, I do:

		m_maxTokenIndex = 0;
		m_minTokenIndex = Integer.MAX_VALUE;

	then the visitor does:

	        public void visit( XQueryAST node, int visitType, int depth ) {
	            if ( visitType == PREVISIT )
	                return;
	
	            int mostMin = node.getMinIndex(),
	                mostMax = node.getMaxIndex();
	            for ( XQueryAST child = (XQueryAST)node.getFirstChild();
	                  child != null;
	                  child = (XQueryAST)child.getNextSibling()
	            ) {
	                final int min = child.getMinIndex();
	                final int max = child.getMaxIndex();
	                if ( min < mostMin )
	                    mostMin = min;
	                if ( max > mostMax )
	                    mostMax = max;
	            }
	            node.setMinMaxIndex( mostMin, mostMax );
	        }

	Hence, the token index information propagates bottup-up from
	the terminals to the root of the AST.

	By doing the setting of min/max once during a single-pass
	traversal, I can avoid the inefficient way it was done in
	TrackerASTFactory.

	- Paul



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/