[antlr-interest] "code too large" / can't split AST output into smaller files

Stephen Winnall steve at winnall.ch
Mon Oct 11 04:39:38 PDT 2010 

I've solved the problem. My grammar had a lot of lexical tokens (around 50).  I ended up with a parser rule which had most of those tokens in its start set. I then discovered the '~' operator and rewrote the rule using a complement set. The generated method shrank to about 170 lines of code (from over 6000 previously).

Steve

On 8 Oct 2010, at 21:34, Jim Idle wrote:

> Is this a parser or a lexer rule? Do you have backtracking turned on? Do you
> have a rule that has too many alts within the same rule? You should find
> that right before the method, there is a comment that indicate which piece
> of the grammar generated the code. There are a lot of reasons why this might
> be happening but we can't really help without the grammar file to reference.
> Or at least the  comments before the method and the method name. See where
> the method is called from as that will also help. Open the generated code in
> Netbeans.
> 
> Jim
> 
>> -----Original Message-----
>> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
>> bounces at antlr.org] On Behalf Of Stephen Winnall
>> Sent: Friday, October 08, 2010 11:57 AM
>> To: Edson Tirelli
>> Cc: antlr-interest at antlr.org
>> Subject: Re: [antlr-interest] "code too large" / can't split AST output
> into
>> smaller files
>> 
>> So, I've split my grammar up, which caused a number of knock-on errors,
> but
>> now Antlr does its thing again without complaining.
>> 
>> However, even after moving some parser rules into an imported file, I
> still
>> get the "code too large". So I suspect Edson's "obvious" statement
> indicates
>> the source of the problem. The error report is:
>> 
>> 	/Users/steve/Work/Applications/Xxxx/target/generated-
>> sources/antlr3/org/vimia/xxxx/antlr/Root.java:[4701,19] code too large
>> 
>> Closer observation reveals that the error points at a method which starts
> at
>> line 4701 and ends on line 11091. I can well imagine that that may be over
> 64K
>> :-)
>> 
>> But I can see which rule it belongs to, so I can try and refactor the
> rule.
>> 
>> Thanks for all your help.
>> 
>> Steve
>> 
>> On 8 Oct 2010, at 18:22, Edson Tirelli wrote:
>> 
>>>  Also, I might be stating the obvious, but I lost some time on this
>>> recently: usually the "code too large" refers to a method that is over
>>> the 64k limit, and not due to the size of the whole class. If that is
>>> your case, it does not help to split the grammar, as the offending
>>> method will sill be there.
>>> 
>>>  In my case I had to rewrite the offending grammar rule to reduce the
>>> size of the generated method. Once I did that, I was able to go back
>>> to a non-composite grammar, that has much better tooling support.
>>> 
>>>  Hope it helps,
>>>    Edson
>>> 
>>> 2010/10/8 Jim Idle <jimi at temporal-wave.com>:
>>>> I don't know why you are seeing a difference other than you are not
>>>> compiling what you think you are in Works, or a different version of
>>>> ABTLR is in use between the two systems. Anyway, you are correct to
>>>> use Maven, and should be using version 3.2.1 at the moment.
>>>> 
>>>> Importing of tree walking and tree generating grammars works fine,
>>>> just remember that the order of the statements is a bit fragile in
>>>> 3.2 and so you have to get the statements in the correct order. Your
>>>> grammar files should be in the antlr3/com/my/package/r.g order and
>>>> your includes in the antlr3/imports directory. Then your importing
>> grammar should be:
>>>> 
>>>> parser grammar x;
>>>> options {}
>>>> import i1, i2, i3, i4, etc;
>>>> tokens {}
>>>> @parser::...
>>>> 
>>>> rule1 : ....
>>>> 
>>>> 
>>>> The grammars you import are:
>>>> 
>>>> parser grammar myimport;
>>>> 
>>>> rulenm: ...;
>>>> 
>>>> 
>>>> Without you including any specific errors or your grammar files, it
>>>> is difficult to see what else you are doing wrong, if anything.
>>>> 
>>>> Jim
>>>> 
>>>>> -----Original Message-----
>>>>> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
>>>>> bounces at antlr.org] On Behalf Of Stephen Winnall
>>>>> Sent: Friday, October 08, 2010 6:45 AM
>>>>> To: antlr-interest at antlr.org
>>>>> Subject: [antlr-interest] "code too large" / can't split AST output
>>>>> into
>>>> smaller
>>>>> files
>>>>> 
>>>>> I'm using Antlr 3.2 with Java 1.6.0_20 under Mac OS X 10.6.4.
>>>>> 
>>>>> I have a composite grammar which generates AST trees. I can run and
>>>>> debug the grammar from AntlrWorks 1.4 without any problem.
>>>>> 
>>>>> However, my grammar is part of a larger project, which I am
>>>>> developing
>>>> with
>>>>> NetBeans 6.9.1 and Maven, using the antlr3-maven-plugin. When I try
>>>>> to build my project, the grammar file gets translated into Java
>>>>> alright, but
>>>> the
>>>>> resultant Java parser file causes a "code too large" error.
>>>>> 
>>>>> I don't understand why AntlrWorks can compile the grammar but
>>>>> NetBeans cannot. Can anyone enlighten me? The generated parser file
>>>>> is 13756 lines long and the "code too large" error under NetBeans
>>>>> appears on line 7204,
>>>> so
>>>>> there seems to be a fundamental difference in what NetBeans/Maven
>>>>> and AntlrWorks are doing.
>>>>> 
>>>>> I have tried to split the grammar into smaller bits to work round
>>>>> this
>>>> problem.
>>>>> I took some parser rules and put them into a different parser
>>>>> grammar, which I imported into the main one. However, nearly all my
>>>>> parser rules include "-> ^( ... )" to generate AST stuff. Antlr
>>>>> complains when it sees
>>>> a "^("
>>>>> in the imported file. I have included "output=AST;" in the options
>>>>> for the imported file.
>>>>> 
>>>>> None of the examples of composite grammars that I have been able to
>>>>> find in the net deal with AST output from the imported file. Is
>>>>> there some
>>>> trick I
>>>>> should be aware of?
>>>>> 
>>>>> Regards
>>>>> Steve
>>>>> --
>>>>> This message has been scanned for viruses and dangerous content by
>>>>> MailScanner, and is believed to be clean.
>>>>> 
>>>>> 
>>>>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>>>>> Unsubscribe:
>>>>> http://www.antlr.org/mailman/options/antlr-interest/your-
>>>>> email-address
>>>> 
>>>> 
>>>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>>>> Unsubscribe:
>>>> http://www.antlr.org/mailman/options/antlr-interest/your-email-addres
>>>> s
>>>> 
>>> 
>>> 
>>> 
>>> --
>>>  Edson Tirelli
>>>  JBoss Drools Core Development
>>>  JBoss by Red Hat @ www.jboss.com
>>> 
>>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>>> Unsubscribe:
>>> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>>> 
>>> --
>>> This message has been scanned for viruses and dangerous content by
>>> MailScanner, and is believed to be clean.
>>> 
>> 
>> 
>> --
>> This message has been scanned for viruses and dangerous content by
>> MailScanner, and is believed to be clean.
>> 
>> 
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
>> email-address
> 
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
> 
> -- 
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
> 


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

More information about the antlr-interest mailing list