[antlr-interest] Addendum: Trying to get an AST in C#. Not working - could use a sanity check.

Sun May 6 17:52:43 PDT 2012

Also, I tried only libraries and tools from
antlr-dotnet-csharpbootstrap-3.4.1.9004.7z, per advice from
http://www.antlr.org/pipermail/antlr-interest/2012-April/044370.html and got
the following:

[With language=CSharp2;]

C:\Users\pbosch\Desktop\antlr-dotnet-csharpbootstrap-3.4.1.9004>Antlr3
Marbles.g

AST.stg 54:23: premature EOF

[With language=CSharp3;]

C:\Users\pbosch\Desktop\antlr-dotnet-csharpbootstrap-3.4.1.9004>Antlr3
Marbles.g

C:\Users\pbosch\Desktop\antlr-dotnet-csharpbootstrap-3.4.1.9004>

For both versions, I used the generated parser & lexer classes, and got the
following errors both times:

The type name 'parse_return' does not exist in the type
'Marbles.MarblesParser'

'Marbles.MarblesParser.parse()' is inaccessible due to its protection level

The name 'HIDDEN' does not exist in the current context ß This was in
generated code.

This was with 

From: Peter Bosch [mailto:pbosch at highpointsoftware.com] 
Sent: Sunday, May 06, 2012 7:19 PM
To: 'antlr-interest at antlr.org'
Subject: Trying to get an AST in C#. Not working - could use a sanity check.

Hi;

I’ll preface with “I am a first time Lexer/Parser generator user, but a long
time software guy.” That, of course, makes me a “noob.”

I’m preparing to build an analysis and round-trip engineering tool in C# for
a 3rd party system with its own syntax. I believe I can create the
foundation of a persistence layer to and from their custom format through an
ANTLR grammar. In order to work out the kinks, I’ve begun a proof-of-concept
on a simple analogue. One thing it needs to do is support an N-ary AST where
N>2, but I think I have reasonable rewrite rules in place to bring that
about.

The challenge has been in getting an AST generated from even that simple
grammar. I have a grammar that passes code generation from ANTLRWorks, and
code (see attached) that seems to have worked for others, in regards to
generating an AST.

It may have to do with the versions of ANTLRWorks and the ANTLR C# runtimes
that I’m linking to. I’ve tried:

ANTLR 3.1.0 C# Runtimes version (3.1.0.39271) & ANTLRWorks 1.4.3 : Odd
partial parsing – skips or duplicates some children.

ANTLR 3.4.1 C# Runtimes version (3.4.1.9004) & ANTLRWorks 1.4.3 : Generated
lexer issues (_channel=HIDDEN ßnot defined) and program issues
(MarblesParser.parse() inaccessible and MarblesParser.parse_return not
defined.)

So my question is - *Should* either of these configurations work? (I’ve seen
posts hinting that certain builds of ANTLR C# runtime are not
fully-featured, and I’m also wondering if there might have been breaking
changes between 3.1.0.x and 3.4.1.x such that my source needs to change to
support a different model.)

If things *should* work as I’ve got ‘em, I’ve inlined the grammar, the
sample input and the Program.cs below in case anyone cares to beat me about
the head and shoulders for some noob-ish-nesses. J 

[Note: A sack of marbles was the initial concept. You’ll see below, we’ve
now graduated to bags of hammers & boxes of rocks. A subject of which I am
currently feeling quite exemplary.]

In any case, thanks in advance for whatever help you can offer.

Regards, (code after the .sig) 

Peter Bosch
Highpoint Software Systems
E-mail : pbosch at highpointsoftware.com
Phone : 262-893-5400
P-Mail : S42 W27451 Oak Grove Lane Waukesha, WI, 53189
Useful and Usable Decision Support & Technical Software

////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////

Marbles.g

////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////

grammar Marbles;

// options, then tokens, then @header then @members.

options {

  language=CSharp2;

  output=AST;

}

tokens {

    ROOT;

}

@header {

    //using System;

}

@parser::namespace { Marbles }

@lexer::namespace { Marbles }

@parser::members { protected void Print(string s){
/*Console.WriteLine("Token: " + s);*/ } }

// Parser Rules.

parse

    :  r=box* EOF -> ^(ROOT $r*)

    ;

color : K_COLOR '=' s=STRINGLITERAL {Print("C");} ->^(K_COLOR $s) 

    ;

weight     : K_WEIGHT '=' w=INTEGER -> ^(K_WEIGHT $w)

    ;

htype : K_HTYPE '=' t=STRINGLITERAL -> ^(K_HTYPE $t)

    ;

rtype : K_RTYPE '=' t=STRINGLITERAL -> ^(K_RTYPE $t)

    ;

rock : 

    '{' K_ROCK c=color w=weight t=rtype '}' {Print("R");} ->^(K_ROCK $c $w
$t)

    ;

hammer     :

    '{' K_HAMMER t=htype? w=weight? '}'  ->^(K_HAMMER $t? $w?)

    ;

box  :

    '[' K_BOX r=rock* ']' {Print("B");} -> ^(K_BOX $r*)

    ;

bag  :

    '[' K_HAMMER h=(hammer)* ']' -> ^(K_HAMMER $h*)

    ;

// Lexer Rules

EOL  : ('\r'? '\n') {$channel=HIDDEN;}

    ;

WS      : (' '|'\t'|'\f'| EOL)+ {$channel=HIDDEN;}

    ;

K_BOX : 'BOX'

    ;

K_BAG : 'BAG'

    ;

K_ROCK     : 'ROCK'

    ;

K_COLOR    : 'COLOR'

    ;

K_HAMMER

    : 'HAMMER'

    ;

K_HTYPE    : 'H_TYPE'

    ;

K_RTYPE    : 'R_TYPE'

    ;

K_WEIGHT

    : 'WEIGHT'

    ;

STRINGLITERAL

    : '"' (~('"' | '\\'))* '"'

    ;

INTEGER    : ('0'..'9')+

    ;

COMMENT

    : '//' ~('\n'|'\r')* '\r'? '\n' {Skip();} //{$channel=HIDDEN;}

    | '/*' ( options {greedy=false;} : . )* '*/' {Skip();}
//{$channel=HIDDEN;}

    ;

////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////

SampleMarbles.txt

////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////

/*[BAG 

{ HAMMER H_TYPE="Claw";} // Bob's your uncle.

{ HAMMER H_TYPE="Ball Peen" WEIGHT=12 ; }

]*/

[BOX

{ ROCK COLOR="Blue" WEIGHT=41 R_TYPE="Schist" }

{ ROCK COLOR="Yellow" WEIGHT=31 R_TYPE="Sulfur" }

{ ROCK COLOR="Black" WEIGHT=21 R_TYPE="Quartz" }

]

[BOX

{ ROCK COLOR="Blue" WEIGHT=42 R_TYPE="Schist" }

{ ROCK COLOR="Yellow" WEIGHT=32 R_TYPE="Sulfur" }

{ ROCK COLOR="Black" WEIGHT=22 R_TYPE="Quartz" }

]

[BOX

{ ROCK COLOR="Blue" WEIGHT=43 R_TYPE="Schist" }

{ ROCK COLOR="Yellow" WEIGHT=33 R_TYPE="Sulfur" }

{ ROCK COLOR="Black" WEIGHT=23 R_TYPE="Quartz" }

]

////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////

Program.cs

////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////

using System;

using Antlr.Runtime;

using Antlr.Runtime.Tree;

//using Antlr.StringTemplate;

using System.IO;

// http://stackoverflow.com/questions/4396080/antlr-3-3-c-sharp-tutorials

//
http://stackoverflow.com/questions/4396080/antlr-3-3-c-sharp-tutorials/43977
99#4397799

namespace Marbles {

    class MainClass {

        public static void Preorder(ITree Tree, int Depth) {

            if (Tree == null) {

                return;

            }

            for (int i = 0; i < Depth; i++) {

                Console.Write("  ");

            }

            Console.WriteLine(Tree);

            for (int i = 0; i < Tree.ChildCount; i++) {

                Preorder(Tree.GetChild(i), Depth + 1);

            }

        }

        public static void Main(string[] args) {

            string s = File.ReadAllText(args[0]);

            ANTLRStringStream Input = new ANTLRStringStream(s);

            MarblesLexer Lexer = new MarblesLexer(Input);

            CommonTokenStream Tokens = new CommonTokenStream(Lexer);

            MarblesParser Parser = new MarblesParser(Tokens);

            MarblesParser.parse_return ParseReturn = Parser.parse();

            CommonTree Tree = (CommonTree)ParseReturn.Tree;

            Preorder(Tree, 0); 

        }

    }

}