[antlr-interest] Using the python output with ANTLR3

[Dahlhoff Achim (CM-AI/PJ-VW37)]

Hello again...

I now tried using the antlr.jar package and the Java grammar for python from the examples package. Now both the generator and the generated parser run without warnings or errors. Thanks a lot for the hint.

Achim Dahlhoff.


> -----Ursprüngliche Nachricht-----
> Von: Benjamin Niemann [mailto:pink at odahoda.de]
> Gesendet: Donnerstag, 19. Februar 2009 20:15
> An: Dahlhoff Achim (CM-AI/PJ-VW37)
> Cc: antlr-interest at antlr.org
> Betreff: Re: [antlr-interest] Using the python output with ANTLR3
>
> Hi,
>
> On Thu, Feb 19, 2009 at 4:10 PM, Dahlhoff Achim (CM-AI/PJ-VW37)
> <Achim.Dahlhoff at de.bosch.com> wrote:
> > Hi all.
> >
> > I just signed up on this list after contacting Mr. Niemann
> about the antlr Python binding. He suggested checking here
> for a little help.
> >
> > To transform a large bunch of java interface files to
> classes and stubs for a (broad) Java-to-C interface, I would
> like to generate all of the wrapper code and tables needed
> for JNI using a code-generator written in Python (I have more
> experience in python and think it es better suited for
> creating text outputs).
> >
> > When I launch ANTLR 3.1.1 with the Python output option on
> the Java 1.5 grammar file (from the antlr website), I am
> experiencing some problems.
> >
> > First of all, antlr outputs an internal error and some
> warnings. Then, the generated python files contain a few
> lines of code, which are more java than python and won't
> compile without some manual fixing. And when I'm done with
> that, the generated parser won't work... :-(
> >
> > Am I missing something here?
> >
> >
> >
> > The output of antlr when taking the Java 1.5 grammer and
> just adding the 'language=Python' option:
> >
> >
> > C:\MIB\ANTLR>java -classpath ./antlrworks-1.2.1.jar
> org.antlr.Tool grammar\Java.g
> >
> > ANTLR Parser Generator  Version 3.1.1
> > error(10):  internal error: eval tree parse error :
> <AST>:0:0: unexpected AST node:
> >
> >
> org.antlr.stringtemplate.language.ActionEvaluator.expr(Unknown Source)
> >
> org.antlr.stringtemplate.language.ActionEvaluator.action(Unkno
> wn Source)
> >
> org.antlr.stringtemplate.language.ASTExpr.evaluateExpression(U
> nknown Source)
> >
> org.antlr.stringtemplate.language.ASTExpr.handleExprOptions(Un
> known Source)
> > org.antlr.stringtemplate.language.ASTExpr.write(Unknown Source)
> > org.antlr.stringtemplate.StringTemplate.write(Unknown Source)
> >
> org.antlr.stringtemplate.language.ConditionalExpr.writeSubTemp
> late(Unknown Sourc
> > e)
> >
> org.antlr.stringtemplate.language.ConditionalExpr.write(Unknow
> n Source)
> > org.antlr.stringtemplate.StringTemplate.write(Unknown Source)
> > org.antlr.stringtemplate.language.ASTExpr.write(Unknown Source)
> >
> org.antlr.stringtemplate.language.ASTExpr.writeAttribute(Unkno
> wn Source)
> >
> org.antlr.stringtemplate.language.ActionEvaluator.action(Unkno
> wn Source)
> > org.antlr.stringtemplate.language.ASTExpr.write(Unknown Source)
> > org.antlr.stringtemplate.StringTemplate.write(Unknown Source)
> > org.antlr.stringtemplate.language.ASTExpr.write(Unknown Source)
> >
> org.antlr.stringtemplate.language.ASTExpr.writeAttribute(Unkno
> wn Source)
> >
> org.antlr.stringtemplate.language.ActionEvaluator.action(Unkno
> wn Source)
> > org.antlr.stringtemplate.language.ASTExpr.write(Unknown Source)
> > org.antlr.stringtemplate.StringTemplate.write(Unknown Source)
> > org.antlr.codegen.CodeGenerator.write(CodeGenerator.java:1275)
> > org.antlr.codegen.Target.genRecognizerFile(Target.java:94)
> >
> org.antlr.codegen.CodeGenerator.genRecognizer(CodeGenerator.java:460)
> > org.antlr.Tool.generateRecognizer(Tool.java:420)
> > org.antlr.Tool.process(Tool.java:287)
> > org.antlr.Tool.main(Tool.java:72)
>
> That looks like a known problem. I don't know what the exact issue was
> (something with the way antlrworks was packages, I think), but using
> the antlr.jar instead of antlrworks.jar (plus stringtemplate and
> antlr2 which are bundled with antlrworks) should fix it.
>
> > Then, the python interpreter stumbles over this:
> >
> >   if not ((t1.getLine() == t2.getLine() && .....
> >
> > (python requires 'and', not '&&'.)
> >
> > or this:
> >
> >   protected boolean enumIsKeyword = true;
> >
> > that looks like java. Python wood need something like
> enumIsKeyword=True
> > Finally this:
> >
> >   if (!enumIsKeyword) _type=Identifier;
> >
> > Here, the ! operator instead of 'not' is used, and the
> colon is missing.
> > It should be:     if not enumIsKeyword: _type=Identifier
>
> As Sam already wrote the action code has to be Python (or whatever the
> target is). There is a Python version of Java.g in the example
> package.
>
> > When I fix these little bugs manually in the antlr output
> and try to feed it with a sample java file, the parser fails:
> >
> >  Traceback (most recent call last):
> >   File "testprog.py", line 59, in ?
> >     main(sys.argv[1:])
> >   File "testprog.py", line 35, in main
> >     parser.compilationUnit()
> >   File "grammar\JavaParser.py", line 361, in compilationUnit
> >     alt8 = self.dfa8.predict(self.input)
> >   File "C:\Dahlhoff\MIB\ANTLRtest\antlr3\dfa.py", line 92,
> in predict
> >     c = input.LA(1)
> >   File "C:\Dahlhoff\MIB\ANTLRtest\antlr3\streams.py", line
> 859, in LA
> >     return self.LT(i).type
> >   File "C:\Dahlhoff\MIB\ANTLRtest\antlr3\streams.py", line
> 801, in LT
> >     self.fillBuffer()
> >   File "C:\Dahlhoff\MIB\ANTLRtest\antlr3\streams.py", line
> 669, in fillBuffer
> >     t = self.tokenSource.nextToken()
> >   File "C:\Dahlhoff\MIB\ANTLRtest\antlr3\recognizers.py",
> line 1133, in nextToken
> >     self._state.tokenStartCharIndex = self.input.index()
> >  TypeError: find/rfind/index/rindex() takes at least 1
> argument (0 given)
>
> That looks like the input of the lexer is a string. The input for the
> lexer has to be one of the CharStream classes.
>
>
> -Ben
>