[antlr-interest] 答复: antlr-interest Digest, Vol 93, Issue 15
Martin Zhang
martin.zhang717 at gmail.com
Wed Aug 15 18:16:21 PDT 2012
Hello, I do not want these mails any more, but I forgot my password, can
anyone help?
Thanks,
Martin
-----邮件原件-----
发件人: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] 代表 antlr-interest-request at antlr.
org
发送时间: 2012年8月16日 3:00
收件人: antlr-interest at antlr.org
主题: antlr-interest Digest, Vol 93, Issue 15
Send antlr-interest mailing list submissions to
antlr-interest at antlr.org
To subscribe or unsubscribe via the World Wide Web, visit
http://www.antlr.org/mailman/listinfo/antlr-interest
or, via email, send a message with subject or body 'help' to
antlr-interest-request at antlr.org
You can reach the person managing the list at
antlr-interest-owner at antlr.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of antlr-interest digest..."
Today's Topics:
1. AUTO: Noel Dcosta is out of the office (returning 08/16/2012)
(Noel Dcosta)
2. Re: problem about "the code for the static initializer is
exceeding the 65535 bytes limit" (Francis ANDRE)
3. Re: problem about "the code for the static initializer is
exceeding the 65535 bytes limit" (Zhaohui Yang)
4. big lexer problem (Zhaohui Yang)
5. Re: problem about "the code for the static initializer is
exceeding the 65535 bytes limit" (Francis ANDRE)
6. Re: big lexer problem (Jim Idle)
7. Re: problem about "the code for the static initializer is
exceeding the 65535 bytes limit" (Jim Idle)
----------------------------------------------------------------------
Message: 1
Date: Wed, 15 Aug 2012 04:08:26 +0530
From: Noel Dcosta <nodcosta at in.ibm.com>
Subject: [antlr-interest] AUTO: Noel Dcosta is out of the office
(returning 08/16/2012)
To: antlr-interest at antlr.org
Message-ID:
<OF1E12170B.D9CCF0DA-ON65257A5A.007C5E6F-65257A5A.007C5E6F at in.ibm.com>
Content-Type: text/plain; charset=US-ASCII
I am out of the office until 08/16/2012.
Holiday at ISL
Note: This is an automated response to your message "antlr-interest
Digest, Vol 93, Issue 14" sent on 15/08/2012 0:30:01.
This is the only notification you will receive while this person is away.
------------------------------
Message: 2
Date: Wed, 15 Aug 2012 05:37:13 +0200
From: Francis ANDRE <francis.andre.kampbell at orange.fr>
Subject: Re: [antlr-interest] problem about "the code for the static
initializer is exceeding the 65535 bytes limit"
To: Zhaohui Yang <yezonghui at gmail.com>
Cc: antlr-interest at antlr.org
Message-ID: <502B1969.1070700 at orange.fr>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hi Zhaohui
I am currently working on fixing this issues with antlr3.4... Once I will
have a
proper patch, would you be interested in testing it??
FA
Le 14/08/2012 18:05, Zhaohui Yang a ?crit :
> Hi,
>
> Here we have a big grammar and the generated parser.java got a compilation
> : "the code for the static initializer is exceeding the 65535 bytes
limit".
>
> I've searched the net for a while and found that is a widely known limit
in
> JVM or Javac compiler, and not yet has an option to change it higher.
>
> On the ANTLR side, I found 2 solutions proposed by others, but neither of
> them is totally satisfying:
>
> 1. Seperate the big grammar into 2 *.g files, import one from the other.
> Yes, this removes the compilation error with genereated Java. But
> ANTLRWorks does not support imported grammar well. E.g., I can not
> interpret a rule in the imported grammar, it's simply not in the rule list
> for interpreting. And gunit always fail with rules defined in imported
> grammar.
>
> 2. Modify the generated Java source, seperate the "FOLLOW_xxx_in_yyy"
> constants into several static classes and change references to them
> accordingly.
> This is proposed here -
> http://www.antlr.org/pipermail/antlr-interest/2009-November/036608.html .
> The author of the post actually has a solution into ANTLR source code
(some
> string template). But I can't find the attachment he referred to. And
> that's in 2009, I suspect the fix could be incompatible with current ANTLR
> version.
> Without this fix we have to do the modificaiton manually or write a
> script for that. The script is not that easy.
>
> And we found a 3rd solution by ourself, that also involve changing the
> generated Java:
>
> 3. Remove those FOLLOW_... constant completely, and replace the references
> with "null".
> Surprisingly this works, just no error recovery after this, not a
> problem for us. But we really worry this is unsafe, since it's not
> documented anywhere.
>
> After all, we're looking for any other solution that is easier to apply,
> asumming we'll be constantly changing the grammar and recompile the
parser.
>
> Maybe there is a way to get ANTLRWorks and gunit play well with imported
> grammar?
> Maybe there is already a commandline option for antlr Tool, that can
> genereate FOLLOW_... constants in seperate classes?
> Maybe there is already a commandline option for antlr Tool, that can
> supress FOLLOW_... constants code generation?
>
------------------------------
Message: 3
Date: Wed, 15 Aug 2012 22:17:18 +0800
From: Zhaohui Yang <yezonghui at gmail.com>
Subject: Re: [antlr-interest] problem about "the code for the static
initializer is exceeding the 65535 bytes limit"
To: francis.andre.kampbell at orange.fr
Cc: antlr-interest at antlr.org
Message-ID:
<CAO51=GTn42+SZk0E_0y-KPdXzJ-P_cwU9hjJqkR_u6jT37nc6A at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
It's great someone is already trying a fix. I'd be glad to test your fix
when it's out.
Would you please introduce a bit what kind of fix is that? Is it for
ANTLRWorks or ANTLR tool, is it a command line option for seperating FOLLOW
set or supressing that, or something else?
2012/8/15 Francis ANDRE <francis.andre.kampbell at orange.fr>
> Hi Zhaohui
>
> I am currently working on fixing this issues with antlr3.4... Once I will
> have a proper patch, would you be interested in testing it??
>
> FA
> Le 14/08/2012 18:05, Zhaohui Yang a ?crit :
>
> Hi,
>>
>> Here we have a big grammar and the generated parser.java got a
compilation
>> : "the code for the static initializer is exceeding the 65535 bytes
>> limit".
>>
>> I've searched the net for a while and found that is a widely known limit
>> in
>> JVM or Javac compiler, and not yet has an option to change it higher.
>>
>> On the ANTLR side, I found 2 solutions proposed by others, but neither of
>> them is totally satisfying:
>>
>> 1. Seperate the big grammar into 2 *.g files, import one from the other.
>> Yes, this removes the compilation error with genereated Java. But
>> ANTLRWorks does not support imported grammar well. E.g., I can not
>> interpret a rule in the imported grammar, it's simply not in the rule
list
>> for interpreting. And gunit always fail with rules defined in imported
>> grammar.
>>
>> 2. Modify the generated Java source, seperate the "FOLLOW_xxx_in_yyy"
>> constants into several static classes and change references to them
>> accordingly.
>> This is proposed here -
>> http://www.antlr.org/**pipermail/antlr-interest/2009-**
>>
November/036608.html<http://www.antlr.org/pipermail/antlr-interest/2009-Nove
mber/036608.html>.
>> The author of the post actually has a solution into ANTLR source code
>> (some
>> string template). But I can't find the attachment he referred to. And
>> that's in 2009, I suspect the fix could be incompatible with current
ANTLR
>> version.
>> Without this fix we have to do the modificaiton manually or write a
>> script for that. The script is not that easy.
>>
>> And we found a 3rd solution by ourself, that also involve changing the
>> generated Java:
>>
>> 3. Remove those FOLLOW_... constant completely, and replace the
references
>> with "null".
>> Surprisingly this works, just no error recovery after this, not a
>> problem for us. But we really worry this is unsafe, since it's not
>> documented anywhere.
>>
>> After all, we're looking for any other solution that is easier to apply,
>> asumming we'll be constantly changing the grammar and recompile the
>> parser.
>>
>> Maybe there is a way to get ANTLRWorks and gunit play well with
imported
>> grammar?
>> Maybe there is already a commandline option for antlr Tool, that can
>> genereate FOLLOW_... constants in seperate classes?
>> Maybe there is already a commandline option for antlr Tool, that can
>> supress FOLLOW_... constants code generation?
>>
>>
>
--
Regards,
Yang, Zhaohui
------------------------------
Message: 4
Date: Wed, 15 Aug 2012 23:17:58 +0800
From: Zhaohui Yang <yezonghui at gmail.com>
Subject: [antlr-interest] big lexer problem
To: antlr-interest at antlr.org
Message-ID:
<CAO51=GRXpR8teCYw=D5G-4pGATvOKa9XQhD-N0LFU8vQUW4kyA at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Hi,
I'm having big problem with big generated Lexer.java. Any help appreciated.
The language is COBOL. And I found multiple reasons that the lexer get's
too big:
1. I'm adding semantic predicate into the lexer, to simulate "lexer state"
as in YACC and JavaCC. It's like
PICTURE_STRING: {lexerState==PIXTURE_STATE}?=> blah blah //
matching things like AXX(9).99 after a 'PIC' key word
The lexer without semantic predicates is 18K lines.
When I add predicates to one or two of the lexer rules, it grows to more
than 20K.
When I add a single more, it explodes to more than 60K and ANTLR give up
generating lexer with error: code is too long.
2. COBOL has a LOT of key words, that may explain the original 18K lines.
3. I have tokens referencing other tokens.
I've inlined most of them now, as suggested by others. But the size has
not reduced much.
So the question could be:
1. how to generate smaller lexer without removing semantic predicate?
2. If that's not possible, how to simulate "lexer state" without semantic
predicate?
3. Any other solution?
Thanks.
--
Regards,
Yang, Zhaohui
------------------------------
Message: 5
Date: Wed, 15 Aug 2012 19:14:03 +0200
From: Francis ANDRE <francis.andre.kampbell at orange.fr>
Subject: Re: [antlr-interest] problem about "the code for the static
initializer is exceeding the 65535 bytes limit"
To: Zhaohui Yang <yezonghui at gmail.com>
Cc: antlr-interest at antlr.org
Message-ID: <502BD8DB.3010605 at orange.fr>
Content-Type: text/plain; charset=UTF-8; format=flowed
Le 15/08/2012 16:17, Zhaohui Yang a ?crit :
> It's great someone is already trying a fix. I'd be glad to test your
> fix when it's out.
>
> Would you please introduce a bit what kind of fix is that? Is it for
> ANTLRWorks or ANTLR tool, is it a command line option for seperating
> FOLLOW set or supressing that, or something else?
The 64K syndrone is a pure Java problem due to the constraint that the
JVM does not support static initializer greater than 64K -- shame on it
--. Thus if you look to the generated lexer and parser, you will see
certainly a lot of DFA classes, each of them having some static
initializer values. The point is that the sum of the static initializer
of all those DFAs is greater than 64K while the static initialization of
each DFA is somewhat small or in most of case les than 64K. Thus, one
solution is to extract all those DFAs classes and put them outside the
lexer or the parser in fixed directories like the following pattern:
Let <grammar> the directory of the grammar to generate, then all the
generated DFAs will go in
for the lexer's DFAs: package <grammar>.lexer;
for the parser's DAFs: package <grammar>.parser;
and the reference of all those DFAs will be
in the lexer: import <grammar>.lexer.*;
in the parser import <grammar>.parser.*;
But hold on, the fix has to be approved by Terr and I did not yet submit
it. It need to pass all unit tests of the ANTLR3.4 and I am working on
it... there is a real challenge getting the parser/lexer compiled for
java code generated without a package...; and all those unit tests are
producing java parser/lexer at the top level directory.
>
> 2012/8/15 Francis ANDRE <francis.andre.kampbell at orange.fr
> <mailto:francis.andre.kampbell at orange.fr>>
>
> Hi Zhaohui
>
> I am currently working on fixing this issues with antlr3.4... Once
> I will have a proper patch, would you be interested in testing it??
>
> FA
> Le 14/08/2012 18:05, Zhaohui Yang a ?crit :
>
> Hi,
>
> Here we have a big grammar and the generated parser.java got a
> compilation
> : "the code for the static initializer is exceeding the 65535
> bytes limit".
>
> I've searched the net for a while and found that is a widely
> known limit in
> JVM or Javac compiler, and not yet has an option to change it
> higher.
>
> On the ANTLR side, I found 2 solutions proposed by others, but
> neither of
> them is totally satisfying:
>
> 1. Seperate the big grammar into 2 *.g files, import one from
> the other.
> Yes, this removes the compilation error with genereated
> Java. But
> ANTLRWorks does not support imported grammar well. E.g., I can not
> interpret a rule in the imported grammar, it's simply not in
> the rule list
> for interpreting. And gunit always fail with rules defined in
> imported
> grammar.
>
> 2. Modify the generated Java source, seperate the
> "FOLLOW_xxx_in_yyy"
> constants into several static classes and change references to
> them
> accordingly.
> This is proposed here -
>
http://www.antlr.org/pipermail/antlr-interest/2009-November/036608.html
> .
> The author of the post actually has a solution into ANTLR
> source code (some
> string template). But I can't find the attachment he referred
> to. And
> that's in 2009, I suspect the fix could be incompatible with
> current ANTLR
> version.
> Without this fix we have to do the modificaiton manually
> or write a
> script for that. The script is not that easy.
>
> And we found a 3rd solution by ourself, that also involve
> changing the
> generated Java:
>
> 3. Remove those FOLLOW_... constant completely, and replace
> the references
> with "null".
> Surprisingly this works, just no error recovery after
> this, not a
> problem for us. But we really worry this is unsafe, since it's not
> documented anywhere.
>
> After all, we're looking for any other solution that is easier
> to apply,
> asumming we'll be constantly changing the grammar and
> recompile the parser.
>
> Maybe there is a way to get ANTLRWorks and gunit play well
> with imported
> grammar?
> Maybe there is already a commandline option for antlr Tool,
> that can
> genereate FOLLOW_... constants in seperate classes?
> Maybe there is already a commandline option for antlr Tool,
> that can
> supress FOLLOW_... constants code generation?
>
>
>
>
>
> --
> Regards,
>
> Yang, Zhaohui
>
------------------------------
Message: 6
Date: Wed, 15 Aug 2012 11:32:08 -0700
From: Jim Idle <jimi at temporal-wave.com>
Subject: Re: [antlr-interest] big lexer problem
To: antlr-interest at antlr.org
Message-ID: <d97497be4a0866591621e45230ff9041 at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
This really means that your lexer is too complicated and I suspect that
you are just trying to type in a grammar from a normative spec without
thinking ahead a little (not trying to insult you here). The specs are
usually designed to explain the language/syntax, not necessarily to be
copied straight in to a parser grammar.
You should really post your grammar files to get better help, but
generally you are trying to introduce context/state in to the lexer, which
is not necessary in all but a few cases. For instance, why do you care
about the token type in the lexer if the same pattern is used for two
token types? Take a token that matches a PIC pattern generally, then
verify that the pattern is a good PIC spec when you are walking the tree,
not in the lexer.
On top of this, if you are trying to drive the lexer state from the
parser, then it is very unlikely it will work anyway.
Try to take a step back, and reduce the number of tokens to a minimum,
remove any state that you can, move all the error checking and validation
as far away from the lexer as you can (at the lexer level you have a
minimum context, at the tree walk level you have much more information and
can issue much better errors/warnings).
Next, you don't need a 'fix' for ANTLR. You will find that as you simplify
the grammar and spend time on left factoring the rules, that all/a lot of
your problems will go away. If you still have issues with generated code
size at that point, then you need to start importing grammars and
debugging remotely (do not use the interpreter in ANTLRWorks anyway), not
trying to change the output of ANTLR. The only time I have had to use
imports is for a full TSQL grammar, which is huge because SQL is so
terrible. COBOL is pretty big, but nothing like SQL.
Jim
> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Zhaohui Yang
> Sent: Wednesday, August 15, 2012 8:18 AM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] big lexer problem
>
> Hi,
>
> I'm having big problem with big generated Lexer.java. Any help
> appreciated.
>
> The language is COBOL. And I found multiple reasons that the lexer
> get's too big:
>
> 1. I'm adding semantic predicate into the lexer, to simulate "lexer
> state"
> as in YACC and JavaCC. It's like
>
> PICTURE_STRING: {lexerState==PIXTURE_STATE}?=> blah blah //
> matching things like AXX(9).99 after a 'PIC' key word
>
> The lexer without semantic predicates is 18K lines.
> When I add predicates to one or two of the lexer rules, it grows to
> more than 20K.
> When I add a single more, it explodes to more than 60K and ANTLR
> give up generating lexer with error: code is too long.
>
> 2. COBOL has a LOT of key words, that may explain the original 18K
> lines.
>
> 3. I have tokens referencing other tokens.
> I've inlined most of them now, as suggested by others. But the size
> has not reduced much.
>
> So the question could be:
> 1. how to generate smaller lexer without removing semantic predicate?
> 2. If that's not possible, how to simulate "lexer state" without
> semantic predicate?
> 3. Any other solution?
>
> Thanks.
>
> --
> Regards,
>
> Yang, Zhaohui
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address
------------------------------
Message: 7
Date: Wed, 15 Aug 2012 11:38:54 -0700
From: Jim Idle <jimi at temporal-wave.com>
Subject: Re: [antlr-interest] problem about "the code for the static
initializer is exceeding the 65535 bytes limit"
Cc: antlr-interest at antlr.org
Message-ID: <925ce1df1c75295afe33be1ab5c7d9c9 at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
It does not need a fix. It is the grammar that needs to be improved. The
huge DFAs are indicative of your grammars being overly complicated or poorly
left factored. ANTLR might do better than it does in some cases, and v4 may
well get around a lot of similar issues, but in general, improve your
grammar files.
First, look at the generated DFA. What rule, or combination of rules is
generating this? Start there. Left factor. Simplify. Stop trying to do much
of anything in the lexer other than match the simplest common token set.
Stop trying to impose semantics in the parser ("you can only have at most
two of 'these' here" - push such things in the tree walk, or add semantic
checks (allow any number of 'these', count how many you got, then issue a
semantic error).
Writing good grammars is not easy. In some ways, because it is easy to just
type stuff in and give it a whirl, ANTLR can cause you to shoot yourself in
the foot!
Step back and consider your grammar files. Do you really want a grammar that
generates such huge decision tables? What is going wrong? It usually is not
ANTLR itself.
Jim
> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Francis ANDRE
> Sent: Wednesday, August 15, 2012 10:14 AM
> To: Zhaohui Yang
> Cc: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] problem about "the code for the static
> initializer is exceeding the 65535 bytes limit"
>
> Le 15/08/2012 16:17, Zhaohui Yang a ?crit :
> > It's great someone is already trying a fix. I'd be glad to test your
> > fix when it's out.
> >
> > Would you please introduce a bit what kind of fix is that? Is it for
> > ANTLRWorks or ANTLR tool, is it a command line option for seperating
> > FOLLOW set or supressing that, or something else?
> The 64K syndrone is a pure Java problem due to the constraint that the
> JVM does not support static initializer greater than 64K -- shame on
> it --. Thus if you look to the generated lexer and parser, you will see
> certainly a lot of DFA classes, each of them having some static
> initializer values. The point is that the sum of the static initializer
> of all those DFAs is greater than 64K while the static initialization
> of each DFA is somewhat small or in most of case les than 64K. Thus,
> one solution is to extract all those DFAs classes and put them outside
> the lexer or the parser in fixed directories like the following
> pattern:
>
> Let <grammar> the directory of the grammar to generate, then all the
> generated DFAs will go in
>
> for the lexer's DFAs: package <grammar>.lexer;
> for the parser's DAFs: package <grammar>.parser;
>
> and the reference of all those DFAs will be
> in the lexer: import <grammar>.lexer.*;
> in the parser import <grammar>.parser.*;
>
> But hold on, the fix has to be approved by Terr and I did not yet
> submit it. It need to pass all unit tests of the ANTLR3.4 and I am
> working on it... there is a real challenge getting the parser/lexer
> compiled for java code generated without a package...; and all those
> unit tests are producing java parser/lexer at the top level directory.
> >
> > 2012/8/15 Francis ANDRE <francis.andre.kampbell at orange.fr
> > <mailto:francis.andre.kampbell at orange.fr>>
> >
> > Hi Zhaohui
> >
> > I am currently working on fixing this issues with antlr3.4...
> Once
> > I will have a proper patch, would you be interested in testing
> it??
> >
> > FA
> > Le 14/08/2012 18:05, Zhaohui Yang a ?crit :
> >
> > Hi,
> >
> > Here we have a big grammar and the generated parser.java got
> a
> > compilation
> > : "the code for the static initializer is exceeding the 65535
> > bytes limit".
> >
> > I've searched the net for a while and found that is a widely
> > known limit in
> > JVM or Javac compiler, and not yet has an option to change it
> > higher.
> >
> > On the ANTLR side, I found 2 solutions proposed by others,
> but
> > neither of
> > them is totally satisfying:
> >
> > 1. Seperate the big grammar into 2 *.g files, import one from
> > the other.
> > Yes, this removes the compilation error with genereated
> > Java. But
> > ANTLRWorks does not support imported grammar well. E.g., I
> can not
> > interpret a rule in the imported grammar, it's simply not in
> > the rule list
> > for interpreting. And gunit always fail with rules defined in
> > imported
> > grammar.
> >
> > 2. Modify the generated Java source, seperate the
> > "FOLLOW_xxx_in_yyy"
> > constants into several static classes and change references
> to
> > them
> > accordingly.
> > This is proposed here -
> > http://www.antlr.org/pipermail/antlr-interest/2009-
> November/036608.html
> > .
> > The author of the post actually has a solution into ANTLR
> > source code (some
> > string template). But I can't find the attachment he referred
> > to. And
> > that's in 2009, I suspect the fix could be incompatible with
> > current ANTLR
> > version.
> > Without this fix we have to do the modificaiton manually
> > or write a
> > script for that. The script is not that easy.
> >
> > And we found a 3rd solution by ourself, that also involve
> > changing the
> > generated Java:
> >
> > 3. Remove those FOLLOW_... constant completely, and replace
> > the references
> > with "null".
> > Surprisingly this works, just no error recovery after
> > this, not a
> > problem for us. But we really worry this is unsafe, since
> it's not
> > documented anywhere.
> >
> > After all, we're looking for any other solution that is
> easier
> > to apply,
> > asumming we'll be constantly changing the grammar and
> > recompile the parser.
> >
> > Maybe there is a way to get ANTLRWorks and gunit play well
> > with imported
> > grammar?
> > Maybe there is already a commandline option for antlr Tool,
> > that can
> > genereate FOLLOW_... constants in seperate classes?
> > Maybe there is already a commandline option for antlr Tool,
> > that can
> > supress FOLLOW_... constants code generation?
> >
> >
> >
> >
> >
> > --
> > Regards,
> >
> > Yang, Zhaohui
> >
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address
------------------------------
_______________________________________________
antlr-interest mailing list
antlr-interest at antlr.org
http://www.antlr.org/mailman/listinfo/antlr-interest
End of antlr-interest Digest, Vol 93, Issue 15
**********************************************
More information about the antlr-interest
mailing list