[antlr-interest] (v3) Token aggregation

[Sean Walton]

Terence Parr wrote:
> actually it's the id+=foo that has been fixed already in my version.
>
> Note the error says idx which means it has nothing to do with the rule 
> you provide.
>
> Ter
oops.  Bad snapshot.
The version you are talking about... is it released yet?  Because here 
is a test case that demonstrates the problem:

    grammar Test;
    options {
        output=AST;
        memoize=true;
        backtrack=true;
    }

    tokens {
        PATH;
    }

    program options{ k=1; backtracking=false; }
        :    id+=IDENTIFIER ('.' id+=IDENTIFIER) -> ^(PATH $id+)
        ;
       
    IDENTIFIER
        :    ('a'..'z'|'A'..'Z')+
        ;

Generates:

     >java -classpath
    "antlr-3.0b3.jar;stringtemplate-2.3b9.jar;antlr-2.7.6.jar"
    org.antlr.Tool t.g
    ANTLR Parser Generator   Early Access Version 3.0b3 (July 21, 2006) 
    1989-2006
    0:0: syntax error: assign.types: <AST>:13:50: unexpected AST node: id
    0:0: syntax error: assign.types: <AST>:0:0: expecting EOB, found
    '<end-of-alt>'
    0:0: syntax error: assign.types: <AST>:0:0: unexpected end of subtree
    0:0: syntax error: define: <AST>:13:50: unexpected AST node: id
    0:0: syntax error: define: <AST>:0:0: expecting EOB, found
    '<end-of-alt>'
    0:0: syntax error: define: <AST>:0:0: unexpected end of subtree
    0:0: syntax error: buildnfa: <AST>:13:50: unexpected AST node: id
    internal error: t.g : java.lang.NullPointerException
    org.antlr.tool.NFAFactory.build_AlternativeBlock(NFAFactory.java:373)
    org.antlr.tool.TreeToNFAConverter.block(TreeToNFAConverter.java:779)
    org.antlr.tool.TreeToNFAConverter.rule(TreeToNFAConverter.java:548)
    org.antlr.tool.TreeToNFAConverter.rules(TreeToNFAConverter.java:374)
    org.antlr.tool.TreeToNFAConverter.grammarSpec(TreeToNFAConverter.java:328)
    org.antlr.tool.TreeToNFAConverter.grammar(TreeToNFAConverter.java:188)
    org.antlr.tool.Grammar.createNFAs(Grammar.java:677)
    org.antlr.tool.Grammar.checkAllRulesForLeftRecursion(Grammar.java:1288)
    org.antlr.codegen.CodeGenerator.genRecognizer(CodeGenerator.java:276)
    org.antlr.Tool.processGrammar(Tool.java:320)
    org.antlr.Tool.process(Tool.java:251)
    org.antlr.Tool.main(Tool.java:70)

Deleting the "backtracking=true;" I get:

     >java -classpath
    "antlr-3.0b3.jar;stringtemplate-2.3b9.jar;antlr-2.7.6.jar"
    org.antlr.Tool t.g
    ANTLR Parser Generator   Early Access Version 3.0b3 (July 21, 2006) 
    1989-2006

     >

Putting it back in and removing the '+' in the rewrite rule, I get:

     >java -classpath
    "antlr-3.0b3.jar;stringtemplate-2.3b9.jar;antlr-2.7.6.jar"
    org.antlr.Tool t.g
    ANTLR Parser Generator   Early Access Version 3.0b3 (July 21, 2006) 
    1989-2006

     >

Any ideas?
-Sean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20060805/8919d9a0/attachment.html