[antlr-interest] Antlr 3.2 issues: code too large and "no such attribute: description"

Jim Idle jimi at temporal-wave.com
Wed Aug 18 07:58:18 PDT 2010


A few pointers:

1) Use the complete self-contained jar download and not the string of
individual jars for development, then with the command:

java -Xmx750M -Xincgc -jar C:/antlr/antlr-master-3.2-completejar.jar
-Xconversiontimeout 32000 -debug %*

In a batch file (or script) your grammar generates correctly.

2) You are trying to specify the lexer a little too strongly. The problem
with what you have is that the best you can do error wise with a wrong
character is use the ANY : . {error}; rule to say it is illegal. Ideally
don't allow such huge ranges, but if the language says they are legal then
you have to deal with them of course. 

So where possible you want to allow just about anything you can in an
identifier, then use a semantic check rule that can say "Identifier
'x\u7858yzzz' is illegally named as the character '\u7858'. You just have to
be careful with ambiguities. Perhaps you have to look for the specific
lead-in but then consume until something that must be some other token start
(don't be afraid to use code to do this as it will often be neater than the
generated code for huge tables like this.

Next, don't use ESC : '\' <longlist of alts> for escape codes, as again you
will just get 'error at character 'q' from the lexer which is of no use.
Allow any character to be escaped lexically then check it later where you
can say "The escape sequence \q in string 'aaa\qaaaa'" is not recognized",
which is much more useful.

You can apply this technique to a lot of lexer rules (even hex strings for
instance) and greatly improve the error messages.

3) Take as much code out of the parser action as you can and create a helper
interface, of which your current code gen is an implementer. Pass this in
before parsing starts. Your parser does not look too far away from this, but
instead of 3 or 4 consecutive method calls, you should consider a single
call for a single purpose. Further, if you can do so, other than perhaps
creating a symbol table, don't do anything in the parser except create the
AST, then do a semantic validation  of the AST with good error messages,
then proceed to whatever else the code should do (including syntax
highlighting for instance.) You will have a sequence then where you can
perform syntax highlighting stuff, but also occasionally do semantic checks
(look at how Visual Studio handles C#  for instance).

Hope that is useful to you,

Jim




> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Edson Tirelli
> Sent: Tuesday, August 17, 2010 5:49 PM
> To: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] Antlr 3.2 issues: code too large and "no
such
> attribute: description"
> 
>    Hi Jim,
> 
>    Thanks for the reply. Lets simplify things and forget antlrworks and
the
> code too large for now. The most important for me is to be able to compile
> the grammar with -debug on the command line.
> 
>    On the command line, I am using the following:
> 
> $ java -classpath
>
"antlr-3.2.jar;antlr-runtime-3.2.jar;antlr-2.7.7.jar;stringtemplate-3.2.1.ja
r"
> org.antlr.Tool -debug -lib src/lang src/lang/DRL.g
> 
>      I get the error:
> 
>     [java] error(10):  internal error:
> src/main/resources/org/drools/lang/DRL.g :
> java.util.NoSuchElementException:
> no such attribute: description in template context [orPredicates
notPredicate
> evalPredicate(...)]
> 
>    That seems the same problem reported in
> http://www.antlr.org/jira/browse/ANTLR-378 .
> 
>    My grammar is here:
>
http://anonsvn.jboss.org/repos/labs/labs/jbossrules/branches/etirelli/drool
> s-compiler/src/main/resources/org/drools/lang/DRL.g
> 
>    As I mentioned before, it is fairly complex. Also, there is a ton of
code in
> there to support the code editor, what "polutes" the grammar, so I don't
> expect anyone to understand it. The important thing is it does compile
> without any warnings or errors, except for when I try to use the -debug
flag.
> 
>    Thanks
> 
>    Edson
> 
> 2010/8/17 Jim Idle <jimi at temporal-wave.com>
> 
> > The JIRA you quote is not showing the same problems as the error you
> > quote, hence the patch is unlikely to help. If your grammar is too
> > large with the debug stuff in it then it is possible that its
> > structure is not so good. It is also possible that it is just too big
> > for a single file of course. As you say, you can debug at the java
> > level too - the multiple grammar debugging stuff in ANTLRWorks does
> > not seem to be quite there.
> >
> > Are you using the latest jar download to generate at the command line
> > and not the ANTLR that is embedded in ANTLRWorks? You should really
> > supply the invocations you are using as otherwise we have to guess.
> > You could try generating using the ANTLRWorks jar. However, if the
> > code is too big with debug stuff in it, then there isn't much you can
> > do but start using imports.
> > Also, try increasing the conversion timeout.
> >
> > Jim
> >
> >
> >
> > > -----Original Message-----
> > > From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> > > bounces at antlr.org] On Behalf Of Edson Tirelli
> > > Sent: Tuesday, August 17, 2010 2:11 PM
> > > To: antlr-interest at antlr.org
> > > Subject: [antlr-interest] Antlr 3.2 issues: code too large and "no
> > > such
> > > attribute: description"
> > >
> > >    Hey all,
> > >
> > >    Need some advice. Question:
> > >
> > > * short version: Does anyone tried applying the patch from this
> > > ticket (
> > > http://www.antlr.org/jira/browse/ANTLR-378) on ANTLR 3.2? Is it
> > > still
> > good?
> > >
> > > * long version:
> > >
> > >    I am adding support for full Java expressions to Drools
> > (www.drools.org)
> > > DRL language, and basically my grammar (that was already complex)
> > > now is also twice the size it was. Before, I was using ANTLRWorks
> > > (1.4) to debug
> > my
> > > grammar but now, when I try to debug I get the error:
> > >
> > > "code too large"
> > >
> > >    That is a bit strange to me as the grammar generates just fine
> > > when I
> > use
> > > the command line tool to generate it, but I figured it might be due
> > > to
> > some
> > > additional code ANTLRWorks might be generating for debug purposes.
> > > My search through this list archives and internet found some
> > > comments
> > stating
> > > that the way to go in this case would be to use Composite Grammars,
> > > but
> > the
> > > problem seems to be that neither antlrworks nor the eclipse plugin
> > > have proper support to it. I gave it a quick try and it does
> > > highlight missing
> > rules as
> > > errors in dependent grammars making maintenance a lot harder.
> > >
> > >     So, if I would have to do remote debugging anyway, I tried to do
> > > it
> > with my
> > > current grammar. Although, when I try to generate the grammar with
> > > the "- debug" flag on the command line, I get:
> > >
> > >     "[java] error(10):  internal error:
> > > src/main/resources/org/drools/lang/DRL.g :
> > > java.util.NoSuchElementException:
> > > no such attribute: description in template context [orPredicates
> > notPredicate
> > > evalPredicate(...)]"
> > >
> > >     This is apparently a known bug reported since ANTLR 3.1.1 (
> > > http://www.antlr.org/jira/browse/ANTLR-378) for which a patch was
> > > provided but apparently never applied.
> > >
> > >     Right now I am stuck, since I can not debug neither locally nor
> > remotely, I
> > > see only two options: either I go trial and error working on my
> > > grammar (what can be extremely time consuming for a complex grammar
> > > like
> > > ours) or I try to apply the patch and build my own
> > > StringTemplate/ANTLR binaries and hope that it will work.
> > >
> > >     So, before spending more time on the later, I wanted to check if
> > anyone
> > > else tried that and succeeded, otherwise, I will just go the old way
> > > and
> > debug
> > > the generated code itself. :(
> > >
> > >     Anyone worked around these problems yet? Any advice?
> > >
> > >     Thanks a lot,
> > >        Edson
> > >
> > > --
> > >   Edson Tirelli
> > >   JBoss Drools Core Development
> > >   JBoss by Red Hat @ www.jboss.com
> > >
> > > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > > Unsubscribe:
> > > http://www.antlr.org/mailman/options/antlr-interest/your-
> > > email-address
> >
> >
> > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > Unsubscribe:
> > http://www.antlr.org/mailman/options/antlr-interest/your-email-address
> >
> 
> 
> 
> --
>   Edson Tirelli
>   JBoss Drools Core Development
>   JBoss by Red Hat @ www.jboss.com
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address



More information about the antlr-interest mailing list