[antlr-interest] NPE in Tree Grammar

[J. Talafous]

What am I doing wrong... or have I found a new bug in ANTLR Parser
Generator  Version 3.3 Nov 30, 2010 12:45:30?  My grammars debug OK in
ANTLRWorks IDE, but I am still a novice at ANTLR grammars.

My goal is to parse in a series of (key,value) pairs such as in the
following input:

login where username='MyName'\n

Pretty straightforward.  Stacktrace below.

Here is my grammar:
--------------------------------------------------------------------

grammar Test1;

options {
	language=Java;
	output=AST;
}

tokens {
    LOGIN='login';
    USERNAME='username';
    PASSWORD='password';
	WHERE='where';
}

@header {
package test;
}

@lexer::header{
package test;
}

command
	: sentence (NEWLINE! sentence)* NEWLINE!? EOF!
	{
		System.out.println($command.text);
	};

sentence
	: WS!? verb^ WS! whereClause* WS!?
	{
		System.out.println($sentence.text);
	}
	;

whereClause
	: WHERE^ (WS! whereSubClause)+
	;
	
whereSubClause returns [String value1, String value2]
	: noun^ WS!? '='! WS!? (VALUE) {$value1 = $noun.text; $value2 = $VALUE.text;}
	;

verb
	: LOGIN
	;

noun
	:
	| USERNAME
	| PASSWORD
	;

NEWLINE
	: '\r'? '\n'
	;

WS
	: (' '|'\t'|'\n'|'\r')+
	;

VALUE:
	( '"' (~'"')* '"'
		| '\'' (~'\'')* '\''
	)
	;

Here is my Tree grammar
------------------------------------------------------------

tree grammar Test1Walker;
options {
	tokenVocab=Test1; // use tokens in Test1
	ASTLabelType=CommonTree; // use CommonTree nodes
}

@header {
package test;
}

command
	: ^(LOGIN whereOne)
	{
		System.out.println("MATCH: "+$whereOne.key + " " +$whereOne.value + " ");
	}
	;

whereOne returns [String key, String value]
	: ^(WHERE keyValue)
	{
		$key=$keyValue.key;
		$value=$keyValue.value;
	}
	;

keyValue returns [String key, String value]
	: ^(noun VALUE)
	{
		$key = $noun.text;
		$value = $VALUE.text;
	}
	;

noun
	:
	| USERNAME
	| PASSWORD
	;

Here is the Java method to run it
--------------------------------------------------------------

    public static void main(String[] args) throws
RecognitionException, IOException
    {
        test1();
    }

    private static void test1() throws RecognitionException, IOException
    {
        String s = "login where username='MyName'\n" ;
        byte[] testCases = s.getBytes();
        ANTLRInputStream input = new ANTLRInputStream(new
ByteArrayInputStream(testCases));
        Test1Lexer lexer = new Test1Lexer(input);
        CommonTokenStream tokens = new CommonTokenStream(lexer);
        Test1Parser parser = new Test1Parser(tokens);
        Test1Parser.command_return result = parser.command();
        CommonTree tree = (CommonTree) result.getTree();
        System.out.println("WALK RESULTING TREE");
        System.out.println(tree.toStringTree());
        CommonTreeNodeStream nodes = new CommonTreeNodeStream(tree);
        Test1Walker walker = new Test1Walker(nodes);
        walker.command();
    }

Here is the console output complete with NPE
------------------------------------------

login where username='MyName'

WALK RESULTING TREE
(login (where (username 'MyName')))
Exception in thread "main" java.lang.NullPointerException
	at test.Test1Walker.keyValue(Test1Walker.java:156)
	at test.Test1Walker.whereOne(Test1Walker.java:102)
	at test.Test1Walker.command(Test1Walker.java:57)
	at test.Onyx.test1(Onyx.java:200)
	at test.Onyx.main(Onyx.java:183)


Here is the generated Test1Walker.java
------------------------------------------------
look for LINE 156 below
    // $ANTLR start "keyValue"
    // Test1Walker.g:26:1: keyValue returns [String key, String value]
: ^( noun VALUE ) ;
    public final Test1Walker.keyValue_return keyValue() throws
RecognitionException {
        Test1Walker.keyValue_return retval = new Test1Walker.keyValue_return();
        retval.start = input.LT(1);

        CommonTree VALUE4=null;
        Test1Walker.noun_return noun3 = null;


        try {
            // Test1Walker.g:27:2: ( ^( noun VALUE ) )
            // Test1Walker.g:27:4: ^( noun VALUE )
            {
            pushFollow(FOLLOW_noun_in_keyValue84);
            noun3=noun();

            state._fsp--;


            match(input, Token.DOWN, null);
            VALUE4=(CommonTree)match(input,VALUE,FOLLOW_VALUE_in_keyValue86);

            match(input, Token.UP, null);

 /*LINE 156: */	retval.key = (noun3!=null?(input.getTokenStream().toString(
              input.getTreeAdaptor().getTokenStartIndex(noun3.start),
              input.getTreeAdaptor().getTokenStopIndex(noun3.start))):null);
            		retval.value = (VALUE4!=null?VALUE4.getText():null);
            	

            }

        }
        catch (RecognitionException re) {
            reportError(re);
            recover(input,re);
        }
        finally {
        }
        return retval;
    }
    // $ANTLR end "keyValue"

The NPE comes from the input.getTokenStream().  The variable
input.toString() is giving me this message:
"com.sun.jdi.InvocationException occurred invoking method."

After many hours, I am still baffled.  Any help appreciated and thanks
for your time.

Joe