[antlr-interest] Antlr 3.2 issues: code too large and "no such attribute: description"

Jim Idle jimi at temporal-wave.com
Wed Aug 18 14:29:37 PDT 2010 

The maven jars are in fact the same as I use maven to build the complete
jar. So perhaps you are not using the same dependencies as the maven built
single jar? 

Japanese identifiers are fine of course, but it is more the complicated DFAs
that will be generated to handle all the options that youcan try to close
down a bit then use semantic checks.

Good luck :-)

Jim

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Edson Tirelli
> Sent: Wednesday, August 18, 2010 12:37 PM
> To: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] Antlr 3.2 issues: code too large and "no
such
> attribute: description"
> 
>    Hi Jim,
> 
>    Thank you a lot for your tips. It was very helpful.
> 
>    A few comments:
> 
> 1. I actually use maven and the project dependencies to generate the
> grammars. The command line was just a "simplified" example. Now I am
> curious why the separate jars still give me the same error when I try to
> generate the grammar, even using your parameters, but when I use the
> "master jar" it works fine?!
> 
>    Using the master jar is a bit complicate as Drools is an open source
project
> and maven relies on the public maven repositories, so ideally, ANTLR
should
> distribute to maven repos jars that work the same as the ones downloaded
> from the project page. But anyway, for my development here I will use the
> master jar for now.
> 
> 2. Regarding the unicode ranges, believe or not, it is a requirement that
users
> are able to create identifiers using oriental characters, since lots of
> customers/users are Japanese. Regarding the ESC, thanks for the tip, I
will try
> to adapt my grammar.
> 
> 3. Thanks. As now I am getting code too large on the command line as well
> when generating the grammar with debug info, I will refactor and split the
> grammar into a Composite grammar. I will try to reduce the code on parser
> actions as you suggested.
> 
>    Thanks a lot,
>       Edson
> 
> 
> 2010/8/18 Jim Idle <jimi at temporal-wave.com>
> 
> > A few pointers:
> >
> > 1) Use the complete self-contained jar download and not the string of
> > individual jars for development, then with the command:
> >
> > java -Xmx750M -Xincgc -jar C:/antlr/antlr-master-3.2-completejar.jar
> > -Xconversiontimeout 32000 -debug %*
> >
> > In a batch file (or script) your grammar generates correctly.
> >
> > 2) You are trying to specify the lexer a little too strongly. The
> > problem with what you have is that the best you can do error wise with
> > a wrong character is use the ANY : . {error}; rule to say it is
> > illegal. Ideally don't allow such huge ranges, but if the language
> > says they are legal then you have to deal with them of course.
> >
> > So where possible you want to allow just about anything you can in an
> > identifier, then use a semantic check rule that can say "Identifier
> > 'x\u7858yzzz' is illegally named as the character '\u7858'. You just
> > have to be careful with ambiguities. Perhaps you have to look for the
> > specific lead-in but then consume until something that must be some
> > other token start (don't be afraid to use code to do this as it will
> > often be neater than the generated code for huge tables like this.
> >
> > Next, don't use ESC : '\' <longlist of alts> for escape codes, as
> > again you will just get 'error at character 'q' from the lexer which is
of no
> use.
> > Allow any character to be escaped lexically then check it later where
> > you can say "The escape sequence \q in string 'aaa\qaaaa'" is not
> > recognized", which is much more useful.
> >
> > You can apply this technique to a lot of lexer rules (even hex strings
> > for
> > instance) and greatly improve the error messages.
> >
> > 3) Take as much code out of the parser action as you can and create a
> > helper interface, of which your current code gen is an implementer.
> > Pass this in before parsing starts. Your parser does not look too far
> > away from this, but instead of 3 or 4 consecutive method calls, you
> > should consider a single call for a single purpose. Further, if you
> > can do so, other than perhaps creating a symbol table, don't do
> > anything in the parser except create the AST, then do a semantic
> > validation  of the AST with good error messages, then proceed to
> > whatever else the code should do (including syntax highlighting for
> > instance.) You will have a sequence then where you can perform syntax
> > highlighting stuff, but also occasionally do semantic checks (look at
> > how Visual Studio handles C#  for instance).
> >
> > Hope that is useful to you,
> >
> > Jim
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> > > bounces at antlr.org] On Behalf Of Edson Tirelli
> > > Sent: Tuesday, August 17, 2010 5:49 PM
> > > To: antlr-interest at antlr.org
> > > Subject: Re: [antlr-interest] Antlr 3.2 issues: code too large and
> > > "no
> > such
> > > attribute: description"
> > >
> > >    Hi Jim,
> > >
> > >    Thanks for the reply. Lets simplify things and forget antlrworks
> > > and
> > the
> > > code too large for now. The most important for me is to be able to
> > compile
> > > the grammar with -debug on the command line.
> > >
> > >    On the command line, I am using the following:
> > >
> > > $ java -classpath
> > >
> >
> > "antlr-3.2.jar;antlr-runtime-3.2.jar;antlr-2.7.7.jar;stringtemplate-3.
> > 2.1.ja
> > r"
> > > org.antlr.Tool -debug -lib src/lang src/lang/DRL.g
> > >
> > >      I get the error:
> > >
> > >     [java] error(10):  internal error:
> > > src/main/resources/org/drools/lang/DRL.g :
> > > java.util.NoSuchElementException:
> > > no such attribute: description in template context [orPredicates
> > notPredicate
> > > evalPredicate(...)]
> > >
> > >    That seems the same problem reported in
> > > http://www.antlr.org/jira/browse/ANTLR-378 .
> > >
> > >    My grammar is here:
> > >
> > http://anonsvn.jboss.org/repos/labs/labs/jbossrules/branches/etirelli/
> > drool
> > > s-compiler/src/main/resources/org/drools/lang/DRL.g
> > >
> > >    As I mentioned before, it is fairly complex. Also, there is a ton
> > > of
> > code in
> > > there to support the code editor, what "polutes" the grammar, so I
> > > don't expect anyone to understand it. The important thing is it does
> > > compile without any warnings or errors, except for when I try to use
> > > the -debug
> > flag.
> > >
> > >    Thanks
> > >
> > >    Edson
> > >
> > > 2010/8/17 Jim Idle <jimi at temporal-wave.com>
> > >
> > > > The JIRA you quote is not showing the same problems as the error
> > > > you quote, hence the patch is unlikely to help. If your grammar is
> > > > too large with the debug stuff in it then it is possible that its
> > > > structure is not so good. It is also possible that it is just too
> > > > big for a single file of course. As you say, you can debug at the
> > > > java level too - the multiple grammar debugging stuff in
> > > > ANTLRWorks does not seem to be quite there.
> > > >
> > > > Are you using the latest jar download to generate at the command
> > > > line and not the ANTLR that is embedded in ANTLRWorks? You should
> > > > really supply the invocations you are using as otherwise we have to
> guess.
> > > > You could try generating using the ANTLRWorks jar. However, if the
> > > > code is too big with debug stuff in it, then there isn't much you
> > > > can do but start using imports.
> > > > Also, try increasing the conversion timeout.
> > > >
> > > > Jim
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> > > > > bounces at antlr.org] On Behalf Of Edson Tirelli
> > > > > Sent: Tuesday, August 17, 2010 2:11 PM
> > > > > To: antlr-interest at antlr.org
> > > > > Subject: [antlr-interest] Antlr 3.2 issues: code too large and
> > > > > "no such
> > > > > attribute: description"
> > > > >
> > > > >    Hey all,
> > > > >
> > > > >    Need some advice. Question:
> > > > >
> > > > > * short version: Does anyone tried applying the patch from this
> > > > > ticket (
> > > > > http://www.antlr.org/jira/browse/ANTLR-378) on ANTLR 3.2? Is it
> > > > > still
> > > > good?
> > > > >
> > > > > * long version:
> > > > >
> > > > >    I am adding support for full Java expressions to Drools
> > > > (www.drools.org)
> > > > > DRL language, and basically my grammar (that was already
> > > > > complex) now is also twice the size it was. Before, I was using
> > > > > ANTLRWorks
> > > > > (1.4) to debug
> > > > my
> > > > > grammar but now, when I try to debug I get the error:
> > > > >
> > > > > "code too large"
> > > > >
> > > > >    That is a bit strange to me as the grammar generates just
> > > > > fine when I
> > > > use
> > > > > the command line tool to generate it, but I figured it might be
> > > > > due to
> > > > some
> > > > > additional code ANTLRWorks might be generating for debug
> purposes.
> > > > > My search through this list archives and internet found some
> > > > > comments
> > > > stating
> > > > > that the way to go in this case would be to use Composite
> > > > > Grammars, but
> > > > the
> > > > > problem seems to be that neither antlrworks nor the eclipse
> > > > > plugin have proper support to it. I gave it a quick try and it
> > > > > does highlight missing
> > > > rules as
> > > > > errors in dependent grammars making maintenance a lot harder.
> > > > >
> > > > >     So, if I would have to do remote debugging anyway, I tried
> > > > > to do it
> > > > with my
> > > > > current grammar. Although, when I try to generate the grammar
> > > > > with the "- debug" flag on the command line, I get:
> > > > >
> > > > >     "[java] error(10):  internal error:
> > > > > src/main/resources/org/drools/lang/DRL.g :
> > > > > java.util.NoSuchElementException:
> > > > > no such attribute: description in template context [orPredicates
> > > > notPredicate
> > > > > evalPredicate(...)]"
> > > > >
> > > > >     This is apparently a known bug reported since ANTLR 3.1.1 (
> > > > > http://www.antlr.org/jira/browse/ANTLR-378) for which a patch
> > > > > was provided but apparently never applied.
> > > > >
> > > > >     Right now I am stuck, since I can not debug neither locally
> > > > > nor
> > > > remotely, I
> > > > > see only two options: either I go trial and error working on my
> > > > > grammar (what can be extremely time consuming for a complex
> > > > > grammar like
> > > > > ours) or I try to apply the patch and build my own
> > > > > StringTemplate/ANTLR binaries and hope that it will work.
> > > > >
> > > > >     So, before spending more time on the later, I wanted to
> > > > > check if
> > > > anyone
> > > > > else tried that and succeeded, otherwise, I will just go the old
> > > > > way and
> > > > debug
> > > > > the generated code itself. :(
> > > > >
> > > > >     Anyone worked around these problems yet? Any advice?
> > > > >
> > > > >     Thanks a lot,
> > > > >        Edson
> > > > >
> > > > > --
> > > > >   Edson Tirelli
> > > > >   JBoss Drools Core Development
> > > > >   JBoss by Red Hat @ www.jboss.com
> > > > >
> > > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > > > > Unsubscribe:
> > > > > http://www.antlr.org/mailman/options/antlr-interest/your-
> > > > > email-address
> > > >
> > > >
> > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > > > Unsubscribe:
> > > > http://www.antlr.org/mailman/options/antlr-interest/your-email-add
> > > > ress
> > > >
> > >
> > >
> > >
> > > --
> > >   Edson Tirelli
> > >   JBoss Drools Core Development
> > >   JBoss by Red Hat @ www.jboss.com
> > >
> > > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > > Unsubscribe:
> > > http://www.antlr.org/mailman/options/antlr-interest/your-
> > > email-address
> >
> >
> > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > Unsubscribe:
> > http://www.antlr.org/mailman/options/antlr-interest/your-email-address
> >
> 
> 
> 
> --
>   Edson Tirelli
>   JBoss Drools Core Development
>   JBoss by Red Hat @ www.jboss.com
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address

More information about the antlr-interest mailing list