[antlr-interest] (v3) Token aggregation

[Sean Walton]

I found that the "$id+" is not available in the backtracking mode.  eeek.
-sean

Sean Walton wrote:
> With a grammar such as:
>
> path
>    : id+=IDENTIFIER ('.' id+=IDENTIFIER) -> ^(PATH $id+)
>    ;
>
> Are there option incompatibilities?  Currently, when I use the $id+ in 
> the AST section, I get:
>
> ANTLR Parser Generator   Early Access Version 3.0b3 (July 21, 2006)  
> 1989-2006
> 0:0: syntax error: assign.types: <AST>:141:55: unexpected AST node: idx
> 0:0: syntax error: assign.types: <AST>:0:0: expecting EOB, found 
> '<end-of-alt>'
> 0:0: syntax error: assign.types: <AST>:0:0: unexpected end of subtree
> 0:0: syntax error: define: <AST>:141:55: unexpected AST node: idx
> 0:0: syntax error: define: <AST>:0:0: expecting EOB, found '<end-of-alt>'
> 0:0: syntax error: define: <AST>:0:0: unexpected end of subtree
> 0:0: syntax error: buildnfa: <AST>:141:55: unexpected AST node: idx
> internal error: ANesC.g : java.lang.NullPointerException
> org.antlr.tool.NFAFactory.build_AlternativeBlock(NFAFactory.java:373)
> org.antlr.tool.TreeToNFAConverter.block(TreeToNFAConverter.java:779)
> org.antlr.tool.TreeToNFAConverter.rule(TreeToNFAConverter.java:548)
> org.antlr.tool.TreeToNFAConverter.rules(TreeToNFAConverter.java:374)
> org.antlr.tool.TreeToNFAConverter.grammarSpec(TreeToNFAConverter.java:328) 
>
> org.antlr.tool.TreeToNFAConverter.grammar(TreeToNFAConverter.java:188)
> org.antlr.tool.Grammar.createNFAs(Grammar.java:677)
> org.antlr.tool.Grammar.checkAllRulesForLeftRecursion(Grammar.java:1288)
> org.antlr.codegen.CodeGenerator.genRecognizer(CodeGenerator.java:276)
> org.antlr.Tool.processGrammar(Tool.java:320)
> org.antlr.Tool.process(Tool.java:251)
> org.antlr.Tool.main(Tool.java:70)
>
> When I use $id, it "works" (but I only get the first token in the 
> loop).  The grammar I adopted has "memoize" and "backtrack".  Are 
> these the problem?  Also, what does "memoize" mean (it's not defined 
> in the WIKI)?
>
> -Sean