[antlr-interest] 3.1b1 trouble with a CSharp2/AST example

[Link, Hamilton]

I'm having trouble producing an abstract syntax tree with ANTLR.  Lexing is working fine, parsing without any translation is working as far as I can tell as well.  Parsing using a grammer with rewrite rules in it dies.

Here's what I'm doing.

I scarfed the polynomial example from the web site and stuck it in a SimpleExpressions.g test file:

------

grammar SimpleExpressions;

options { language = CSharp2; output = AST; }

tokens { MULT; } // imaginary token

start: term ('+'^ term)*

;

term: INT ID -> ^(MULT["*"] INT ID)

| INT exp -> ^(MULT["*"] INT exp)

| exp

| INT

| ID

;

exp : ID '^'^ INT

;

ID : 'a'..'z'+ ;

INT : '0'..'9'+ ;

WS : (' '|'\t'|'\r'|'\n')+ {$channel = HIDDEN;} ;

------



I ran the 3.1b1 version of antlr on it, from the command line, and it produced my .cs files for the lexer and parser without any errors:



$ java -cp "./antlr-3.1b1.jar;./stringtemplate-3.1.jar;./antlr-runtime-3.1b1.ja
r;./antlr-2.7.7.jar;./gunit-1.0.2.jar" org.antlr.Tool "C:\Documents and Setting
s\helink\My Documents\Visual Studio 2008\Projects\Examples\ANTLRTest\TestParser
\SimpleExpressions.g"
ANTLR Parser Generator  Version 3.1b1 (May 20, 2008)  1989-2008

$



I built a sample C# application and it compiles just fine (using antlr.runtime, Antlr3.Runtime, Antlr3.Utility, and StringTemplate as references).  My main program.cs file is as follows, and it is built with the SimpleExpressionsLexer.cs and SimpleExpressionsParser.cs.



------

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using Antlr.Runtime;

using Antlr.Runtime.Tree;

using Antlr.Runtime.Debug;

using Antlr.Utility.Tree;

namespace TestParser

{

class Program

{

static void Main(string[] args)

{

string testString = "2x+3x^5";

SimpleExpressionsLexer lexer = new SimpleExpressionsLexer(new ANTLRStringStream(testString));

CommonTokenStream tokens = new CommonTokenStream(lexer);

SimpleExpressionsParser parser = new SimpleExpressionsParser(tokens);

SimpleExpressionsParser.start_return bar = parser.start();

CommonTree t = (CommonTree)bar.Tree;

String result = (t != null) ? t.ToStringTree() : null;

}

}

}

------



When I run it, I get a "RewriteEmptyStreamException was unhandled" error with the message "token INT".  Here is the exception detail:



Antlr.Runtime.Tree.RewriteEmptyStreamException was unhandled
  Message="token INT"
  Source="Antlr3.Runtime"
  StackTrace:
       at Antlr.Runtime.Tree.RewriteRuleElementStream`1._Next()
       at Antlr.Runtime.Tree.RewriteRuleTokenStream.NextNode()
       at SimpleExpressionsParser.term() in C:\Documents and Settings\helink\My Documents\Visual Studio 2008\Projects\Examples\ANTLRTest\TestParser\SimpleExpressionsParser.cs:line 305
       at SimpleExpressionsParser.start() in C:\Documents and Settings\helink\My Documents\Visual Studio 2008\Projects\Examples\ANTLRTest\TestParser\SimpleExpressionsParser.cs:line 104
       at TestParser.Program.Main(String[] args) in C:\Documents and Settings\helink\My Documents\Visual Studio 2008\Projects\Examples\ANTLRTest\TestParser\Program.cs:line 20
       at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException:

in other words it's dying in term() as called by start() as called by my main program, i.e. it's dying trying to parse.  The code around the error (at the break point) in Visual Studio is....



// 10:15: -> ^( MULT[\"*\"] INT ID )

{

// C:\\Documents and Settings\\helink\\My Documents\\Visual Studio 2008\\Projects\\Examples\\ANTLRTest\\TestParser\\SimpleExpressions.g:10:18: ^( MULT[\"*\"] INT ID )

{

object root_1 = (object)adaptor.GetNilNode();

root_1 = (object)adaptor.BecomeRoot((object)adaptor.Create(MULT, "*"), root_1);

adaptor.AddChild(root_1, stream_INT.NextNode());         // <--- It's dying here, in SimpleExpressionsParser.cs

adaptor.AddChild(root_1, stream_ID.NextNode());

adaptor.AddChild(root_0, root_1);

}



If I get rid of the rewrite rules or whatever you call them, the -> and ^ etc. stuff, and regenerate and recompile, it runs fine but of course it doesn't produce a useful AST, it just produces t==null.



Any help getting this simple example to run would be greatly appreciated.  At the moment however I'm suspecting either a bug in the example or in the CSharp2 target, as there's nothing of my own making in the picture (even the main program is cribbed from the web site example code).



My immediate goal is to get parse trees and ASTs generated from fairly simple expression grammars.  My ultimate goal is to build heterogenous node trees and write code to evaluate them, i.e. my ultimate goal is to make a fairly simple expression interpreter.  I think if I can get the polynomial example working I'll be a great deal closer to that goal.



thanks,

hamilton






-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080522/68a2672c/attachment.html