[antlr-interest] Antlr 3.2 issues: code too large and "no such attribute: description"
Edson Tirelli
tirelli at post.com
Wed Aug 18 12:36:32 PDT 2010
Hi Jim,
Thank you a lot for your tips. It was very helpful.
A few comments:
1. I actually use maven and the project dependencies to generate the
grammars. The command line was just a "simplified" example. Now I am curious
why the separate jars still give me the same error when I try to generate
the grammar, even using your parameters, but when I use the "master jar" it
works fine?!
Using the master jar is a bit complicate as Drools is an open source
project and maven relies on the public maven repositories, so ideally, ANTLR
should distribute to maven repos jars that work the same as the ones
downloaded from the project page. But anyway, for my development here I will
use the master jar for now.
2. Regarding the unicode ranges, believe or not, it is a requirement that
users are able to create identifiers using oriental characters, since lots
of customers/users are Japanese. Regarding the ESC, thanks for the tip, I
will try to adapt my grammar.
3. Thanks. As now I am getting code too large on the command line as well
when generating the grammar with debug info, I will refactor and split the
grammar into a Composite grammar. I will try to reduce the code on parser
actions as you suggested.
Thanks a lot,
Edson
2010/8/18 Jim Idle <jimi at temporal-wave.com>
> A few pointers:
>
> 1) Use the complete self-contained jar download and not the string of
> individual jars for development, then with the command:
>
> java -Xmx750M -Xincgc -jar C:/antlr/antlr-master-3.2-completejar.jar
> -Xconversiontimeout 32000 -debug %*
>
> In a batch file (or script) your grammar generates correctly.
>
> 2) You are trying to specify the lexer a little too strongly. The problem
> with what you have is that the best you can do error wise with a wrong
> character is use the ANY : . {error}; rule to say it is illegal. Ideally
> don't allow such huge ranges, but if the language says they are legal then
> you have to deal with them of course.
>
> So where possible you want to allow just about anything you can in an
> identifier, then use a semantic check rule that can say "Identifier
> 'x\u7858yzzz' is illegally named as the character '\u7858'. You just have
> to
> be careful with ambiguities. Perhaps you have to look for the specific
> lead-in but then consume until something that must be some other token
> start
> (don't be afraid to use code to do this as it will often be neater than the
> generated code for huge tables like this.
>
> Next, don't use ESC : '\' <longlist of alts> for escape codes, as again you
> will just get 'error at character 'q' from the lexer which is of no use.
> Allow any character to be escaped lexically then check it later where you
> can say "The escape sequence \q in string 'aaa\qaaaa'" is not recognized",
> which is much more useful.
>
> You can apply this technique to a lot of lexer rules (even hex strings for
> instance) and greatly improve the error messages.
>
> 3) Take as much code out of the parser action as you can and create a
> helper
> interface, of which your current code gen is an implementer. Pass this in
> before parsing starts. Your parser does not look too far away from this,
> but
> instead of 3 or 4 consecutive method calls, you should consider a single
> call for a single purpose. Further, if you can do so, other than perhaps
> creating a symbol table, don't do anything in the parser except create the
> AST, then do a semantic validation of the AST with good error messages,
> then proceed to whatever else the code should do (including syntax
> highlighting for instance.) You will have a sequence then where you can
> perform syntax highlighting stuff, but also occasionally do semantic checks
> (look at how Visual Studio handles C# for instance).
>
> Hope that is useful to you,
>
> Jim
>
>
>
>
> > -----Original Message-----
> > From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> > bounces at antlr.org] On Behalf Of Edson Tirelli
> > Sent: Tuesday, August 17, 2010 5:49 PM
> > To: antlr-interest at antlr.org
> > Subject: Re: [antlr-interest] Antlr 3.2 issues: code too large and "no
> such
> > attribute: description"
> >
> > Hi Jim,
> >
> > Thanks for the reply. Lets simplify things and forget antlrworks and
> the
> > code too large for now. The most important for me is to be able to
> compile
> > the grammar with -debug on the command line.
> >
> > On the command line, I am using the following:
> >
> > $ java -classpath
> >
>
> "antlr-3.2.jar;antlr-runtime-3.2.jar;antlr-2.7.7.jar;stringtemplate-3.2.1.ja
> r"
> > org.antlr.Tool -debug -lib src/lang src/lang/DRL.g
> >
> > I get the error:
> >
> > [java] error(10): internal error:
> > src/main/resources/org/drools/lang/DRL.g :
> > java.util.NoSuchElementException:
> > no such attribute: description in template context [orPredicates
> notPredicate
> > evalPredicate(...)]
> >
> > That seems the same problem reported in
> > http://www.antlr.org/jira/browse/ANTLR-378 .
> >
> > My grammar is here:
> >
> http://anonsvn.jboss.org/repos/labs/labs/jbossrules/branches/etirelli/drool
> > s-compiler/src/main/resources/org/drools/lang/DRL.g
> >
> > As I mentioned before, it is fairly complex. Also, there is a ton of
> code in
> > there to support the code editor, what "polutes" the grammar, so I don't
> > expect anyone to understand it. The important thing is it does compile
> > without any warnings or errors, except for when I try to use the -debug
> flag.
> >
> > Thanks
> >
> > Edson
> >
> > 2010/8/17 Jim Idle <jimi at temporal-wave.com>
> >
> > > The JIRA you quote is not showing the same problems as the error you
> > > quote, hence the patch is unlikely to help. If your grammar is too
> > > large with the debug stuff in it then it is possible that its
> > > structure is not so good. It is also possible that it is just too big
> > > for a single file of course. As you say, you can debug at the java
> > > level too - the multiple grammar debugging stuff in ANTLRWorks does
> > > not seem to be quite there.
> > >
> > > Are you using the latest jar download to generate at the command line
> > > and not the ANTLR that is embedded in ANTLRWorks? You should really
> > > supply the invocations you are using as otherwise we have to guess.
> > > You could try generating using the ANTLRWorks jar. However, if the
> > > code is too big with debug stuff in it, then there isn't much you can
> > > do but start using imports.
> > > Also, try increasing the conversion timeout.
> > >
> > > Jim
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> > > > bounces at antlr.org] On Behalf Of Edson Tirelli
> > > > Sent: Tuesday, August 17, 2010 2:11 PM
> > > > To: antlr-interest at antlr.org
> > > > Subject: [antlr-interest] Antlr 3.2 issues: code too large and "no
> > > > such
> > > > attribute: description"
> > > >
> > > > Hey all,
> > > >
> > > > Need some advice. Question:
> > > >
> > > > * short version: Does anyone tried applying the patch from this
> > > > ticket (
> > > > http://www.antlr.org/jira/browse/ANTLR-378) on ANTLR 3.2? Is it
> > > > still
> > > good?
> > > >
> > > > * long version:
> > > >
> > > > I am adding support for full Java expressions to Drools
> > > (www.drools.org)
> > > > DRL language, and basically my grammar (that was already complex)
> > > > now is also twice the size it was. Before, I was using ANTLRWorks
> > > > (1.4) to debug
> > > my
> > > > grammar but now, when I try to debug I get the error:
> > > >
> > > > "code too large"
> > > >
> > > > That is a bit strange to me as the grammar generates just fine
> > > > when I
> > > use
> > > > the command line tool to generate it, but I figured it might be due
> > > > to
> > > some
> > > > additional code ANTLRWorks might be generating for debug purposes.
> > > > My search through this list archives and internet found some
> > > > comments
> > > stating
> > > > that the way to go in this case would be to use Composite Grammars,
> > > > but
> > > the
> > > > problem seems to be that neither antlrworks nor the eclipse plugin
> > > > have proper support to it. I gave it a quick try and it does
> > > > highlight missing
> > > rules as
> > > > errors in dependent grammars making maintenance a lot harder.
> > > >
> > > > So, if I would have to do remote debugging anyway, I tried to do
> > > > it
> > > with my
> > > > current grammar. Although, when I try to generate the grammar with
> > > > the "- debug" flag on the command line, I get:
> > > >
> > > > "[java] error(10): internal error:
> > > > src/main/resources/org/drools/lang/DRL.g :
> > > > java.util.NoSuchElementException:
> > > > no such attribute: description in template context [orPredicates
> > > notPredicate
> > > > evalPredicate(...)]"
> > > >
> > > > This is apparently a known bug reported since ANTLR 3.1.1 (
> > > > http://www.antlr.org/jira/browse/ANTLR-378) for which a patch was
> > > > provided but apparently never applied.
> > > >
> > > > Right now I am stuck, since I can not debug neither locally nor
> > > remotely, I
> > > > see only two options: either I go trial and error working on my
> > > > grammar (what can be extremely time consuming for a complex grammar
> > > > like
> > > > ours) or I try to apply the patch and build my own
> > > > StringTemplate/ANTLR binaries and hope that it will work.
> > > >
> > > > So, before spending more time on the later, I wanted to check if
> > > anyone
> > > > else tried that and succeeded, otherwise, I will just go the old way
> > > > and
> > > debug
> > > > the generated code itself. :(
> > > >
> > > > Anyone worked around these problems yet? Any advice?
> > > >
> > > > Thanks a lot,
> > > > Edson
> > > >
> > > > --
> > > > Edson Tirelli
> > > > JBoss Drools Core Development
> > > > JBoss by Red Hat @ www.jboss.com
> > > >
> > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > > > Unsubscribe:
> > > > http://www.antlr.org/mailman/options/antlr-interest/your-
> > > > email-address
> > >
> > >
> > > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > > Unsubscribe:
> > > http://www.antlr.org/mailman/options/antlr-interest/your-email-address
> > >
> >
> >
> >
> > --
> > Edson Tirelli
> > JBoss Drools Core Development
> > JBoss by Red Hat @ www.jboss.com
> >
> > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> > email-address
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
--
Edson Tirelli
JBoss Drools Core Development
JBoss by Red Hat @ www.jboss.com
More information about the antlr-interest
mailing list